Data platforms have evolved over years, often decades, into complex, business-critical ecosystems. These environments still support reporting and operations, but they can become harder to scale, adapt and innovate with. The question is no longer whether to modernise, but how to do so without disrupting the business.
In recent work with a global services organisation operating thousands of decentralised operational units and hundreds of upstream source systems, these pressures were not theoretical. Reporting pipelines supported day-to-day decision-making across the business, but the underlying architecture had reached the limits of what incremental scaling could achieve. Ingestion pipelines were becoming congested, with processing taking up to 22 hours per day.
In environments where central data platforms underpin operational and regulatory reporting, extended downtime or cutover failure is not an acceptable risk. For organisations with tightly coupled upstream systems, the cost of disruption outweighs the perceived speed of a full cutover.
A common misconception is that transformation requires a single, large-scale migration. In reality, this ‘big bang’ approach introduces significant risk, particularly where data pipelines underpin daily operations. Many organisations are shifting to phased, incremental strategies, with these delivering continuous value while maintaining stability.
Moving beyond the ‘big bang’ migration
Large-scale cutovers can seem appealing, but they rarely reflect the complexity of real-world data estates. Legacy platforms often integrate hundreds of upstream systems, each with its own structure and dependencies, and migrating everything at once can create bottlenecks, increase risk and delay value.
A more effective approach is to break transformation into manageable components. By isolating and migrating workloads in stages, organisations reduce risk, validate progress and adapt as challenges emerge.
This approach also enables a ‘split-brain’ model, where legacy and modern platforms run in parallel. Critical reporting continues, while new capabilities are introduced incrementally. This ensures the business remains operational throughout the transition.
In practice, this parallel-run model is often the only viable option at scale. Legacy pipelines may continue processing for extended periods while individual workloads are isolated, re-engineered, and validated on the new platform. This allows teams to transition functionality without pausing or re-implementing large portions of downstream reporting.
From proof of concept to production reality
Proof-of-concept (PoC) initiatives often demonstrate the potential of platforms such as Databricks. However, early success can mask the complexity of scaling into production.
This gap commonly emerges because early PoCs focus on the simplest or most controlled parts of the data estate. When teams move into full-scale migration, they encounter greater variability in data structure, data quality, and upstream change, often requiring significant rework of initial designs.
In one large-scale programme, early proof-of-concept work focused on a relatively simple subset of the data estate. While this demonstrated the art of the possible, scaling into production required significant rework as the full complexity of upstream systems and data variability became apparent.
As businesses move beyond PoC, they often face challenges with data volume, variability and integration – what works for a small dataset may require rework at enterprise scale.
Recognising this early, and planning for iteration, is critical. Transformation is an ongoing process, and decisions should reflect cost, value and business impact.
Standardisation as a foundation for scale
A key barrier to scale in legacy environments is the growth of bespoke pipelines. As new sources are added, complexity increases and performance can decline. In large estates, where platforms ingest data from hundreds of independently evolving systems, bespoke development quickly becomes difficult to maintain and costly to extend.
Standardisation is therefore less about architectural elegance and more about operational resilience. It ensures that ingestion and processing logic can adapt as sources are added, changed or retired, without requiring extensive re-engineering each time.
Databricks enables a more scalable approach through metadata-driven ingestion and standardised pipeline frameworks. Instead of building individual pipelines for each source, systems can be configured through metadata and processed via reusable patterns (for example, using Delta Live Tables). This creates a consistent, testable ingestion layer that reduces development effort while improving reliability.
In practice, this approach also addresses a common but often overlooked challenge – data volatility. In real-world environments, source systems do not remain static – data structures evolve, systems are replaced, and some sources may disappear entirely. A centrally governed, metadata-driven first-pass harmonisation layer allows organisations to apply consistent filtering, validation and transformation rules across all sources, regardless of change.
By standardising ingestion and processing in this way, organisations can move from reactive pipeline development to a more scalable, controlled model, supporting both current operational demands and future growth.
Unlocking the full potential of Databricks
Databricks helps organisations to modernise data estates at scale. It supports distributed processing, batch and real-time workloads, and diverse data types, making it well suited to complex environments.
In large-scale environments, platform selection is often driven by the need to automate data processing at high volume, rather than by feature breadth alone. The ability to support distributed processing, standardised pipeline frameworks, and enterprise-grade engineering practices becomes critical as the number of sources and workloads grows.
It also upholds modern engineering practices such as CI/CD and reusable pipeline design, improving both the speed and quality of delivery. Built-in tooling can also accelerate migration and reduce manual effort, supporting faster time to value.
The role of partnership in successful transformation
Technology alone does not deliver transformation. Organisations must also address architecture, delivery models and change.
CGI works with clients to deliver phased, low-risk transformations. We combine Databricks expertise with experience in complex environments. Our approach includes best practice design, parallel processing strategies and controlled transitions from legacy systems.
In complex migrations, delivery is not limited to implementation. Teams often work alongside client engineers, transferring skills and embedding best practices as part of day-to-day delivery. This enables organisations to progressively take ownership of the platform and continue evolving it beyond the initial migration phases.
Capability building is central to this approach. We upskill client teams and work collaboratively to build long-term independence, ensuring that the platform continues to evolve post-migration.
Building a foundation for the future
Data platform modernisation is not an endpoint. It is part of a wider lifecycle. From assessment to optimisation, the goal is to enable advanced analytics, AI and innovation.
Importantly, incremental modernisation does not imply a short or linear journey. In many organisations, transformation remains ongoing, with workloads continuously prioritised based on business value, delivery risk, and cost. Progress is measured by sustained improvement, not by a single point of completion.
By taking an incremental approach, and combining the strengths of Databricks and CGI, businesses can move beyond legacy constraints. They can build a scalable, future-ready data foundation that supports both operational needs and long-term ambition.
Please reach out directly on LinkedIn to discuss how CGI and Databricks can support your organisation's transformation journey.
About this author
Artir Geci
Director of Consulting Expert
Artir Geci is an Alliance Technology Director at CGI, focused on driving growth through the Databricks partnership. Within CGI’s Databricks Alliance Technology Practice, he bridges delivery excellence with alliance strategy, leading go-to-market execution and shaping reusable accelerators to streamline data and AI delivery. With ...
