 |
This is a guest article by CGI's technology partner Microsoft. |
In the 1950s data was stored and moved using punch-cards, this was perhaps the first solution to storing computer generated data. About ten years later magnetic tape took over as preferred medium of storage, and was soon replaced by hard drives and floppy disks.
Storage evolved and allowed storing of larger and larger amounts of data. Handling all these physical mediums quickly became a struggle, so Database Management Systems (DBMS) were invented to handle the storing and transfer of the data, allowing large scale management and utilization of it – and the first data platforms were born.
New more efficient and user friendly technologies popped up and stayed around, mainly relational databases and SQL. These together with the internet spawned large international data platforms during the 90s, and the same fundamentals are still around today.
So what is new today?
With seemingly unlimited storage capacity and bandwidth in the gigabytes, what does a state of the art data platform provide that has not always been around? Let’s take a look at some features a data platform should provide today.
1. End-to-end connectivity
A modern data platform should be able to seamlessly integrate with any data source. Separate solutions for integrations, data transformation, storage, and BI are today available in the same package. One example would be out-of-the box connectors for integrations, requiring minimal manual effort to implement. Having integrations as a part of the data platform is vital to increasing operational efficiency of the platform, decreasing the need to move data around.
2. Scalable and cost-efficient storage
Storing large amounts of data in a database is usually expensive, since it reserves compute capacity that could be utilized elsewhere. Instead, the data in today’s platforms is stored in binary form on less expensive services, such as a data lake. From there the data can be raised into virtual tables using serverless SQL or other on-demand technologies. Being able to explore data is a key feature when it comes to finding new connections and insight – maybe even in retrospect on historically “useless” data. Providing exploratory tools is a must to enable this.
3. Security and data governance
Security is also of central concern, only the right users should have access to the right data. Integrating identity management already on a storage level prevents unauthorized maintainers of the data lake from accessing data that only should be accessible by e.g. upper management.
Governance is another aspect of security – does the organization know what data is located where? With GDPR and other privacy regulation, being able to locate personally identifiable data is a requirement.
4. Machine learning capabilities
Machine learning or AI might be a buzzword in many areas, but when it comes to enriching data, a machine can replace vast amounts of previously expert work. Let’s take predictive maintenance as an example. Data from several sensors are being ingested into binary storage, a baseline for what is normal is determined, and the platform automatically alerts in case the input signals deviate.
5. APIs for data-driven applications
Not only humans should benefit from a data platform. The entire business, applications included, should be able to interact with the same data. This can be solved by creating an API layer that accesses data from the data platform.
6. Low-code accessibility
Having a data platform means nothing if the end-users cannot interact with the data on it. With today’s low-code tools anyone can become a citizen developer - creating own dashboards and even applications. Democratizing data enables employees outside the traditional BI-teams to quickly access relevant data and perhaps even create new innovative applications that would otherwise have remained on a thought-level.
All in all a modern data platform securely brings the right data to its authorized end users in real-time. The users can in turn explore, analyze, and generate new data to be put back into the platform. Applications can seamlessly interact with the platform through APIs, leading to the entire business and its applications having one true view of all data in its ecosystem. Data governance and security tools constantly monitor the data on the platform, providing a holistic view of the data estate.
What took years of development in the early data platforms of the 90s can today be done in days, truly putting the data platform at the center of the entire business.
