A leading Canadian bank modernized its batch processing environment as older Informatica versions and legacy infrastructure are phased out, while addressing increasing volume demands. Working with CGI, the bank migrated workloads from Informatica to Azure Databricks and has jointly built a standardized migration framework to support a consistent, repeatable approach across multiple applications. The bank’s inventory includes multiple tier 1 and tier 2 applications and approximately 15,000–20,000 workflows to be migrated.

25–30%

The modernization has delivered a 25–30% improvement in overall productivity, improved scalability through automatic compute scaling, reduced manual monitoring of batch jobs, and enabled faster fixes and releases through DevOps automation. Following an initial proof of concept covering five applications, the bank expanded the workstream and awarded CGI an additional 20 applications for migration.

Overview

As data volumes grow, banks are under pressure to ensure batch processing can scale reliably and remain supportable as legacy tools and infrastructure reach end-of-support. As part of its legacy modernization initiative, in coordination with Microsoft, a leading bank was modernizing batch ETL workloads by migrating from Informatica to Azure Databricks using a pro-code approach involving Python/PySpark and the DB Lakeflow technology stack. The focus was on workload migration, rather than data migration, while scaling batch applications to meet increasing volume demand. CGI supported this initiative as an existing partner of the bank.

Business need and goals

The bank needed to modernize batch ETL workloads running on Informatica infrastructure to a modern technology stack on Azure and Databricks. Its inventory included multiple tier 1 and tier 2 applications and approximately 15,000 to 20,000 workflows to be migrated from Informatica to Databricks.

The migration also required integration within the batch ecosystem, including scheduling, dependency resolution between upstream and downstream jobs, and integration with multiple data sources. In addition, the modernization effort is intended to enable batch applications to scale in response to the bank’s increasing volume demands.

Key challenges driving the initiative

The bank pursued the modernization as older versions of Informatica, including versions 9 and 10, were being phased out and support was ending. Its legacy environment was also running on very old infrastructure, including older server stacks and Oracle versions, which the bank wanted to phase out.

At the same time, the bank wanted to reduce its reliance on on-premise technology stacks by moving workloads to a cloud-enabled Databricks environment.

A collaborative approach

We provided trained delivery capacity to support lines of business in migrating workloads from Informatica to Databricks, bringing consulting and implementation skills to the initiative.

We also jointly built a framework and library of Python/PySpark functions to standardize migration and align delivery to a consistent pattern. The framework provides a platform to develop and migrate workflows, enabling a repeatable approach across multiple applications. It incorporates standardized ingestion, transformation and target population patterns, along with monitoring, exception tracing and built-in security, driven by JSON-based configuration.

In addition, we developed an AI-based accelerator using LLM/OpenAI concepts to help expedite migrations using the framework and library built with the bank.

Benefits and outcomes

The work established standard patterns that support consistent code structure, faster scaffolding and debugging, and easier long-term maintenance. It also improved scalability by enabling compute to scale automatically when loads are high, reducing the need for manual intervention.

DevOps automation reduced the time required to deploy, retest and promote changes, with deployments automated after code check-in. The code and notebook-based approach also improved documentation and readability compared with dependence on Informatica designer tooling, while providing better alignment with an available talent pool as Informatica skills become less common.

Overall productivity improved by 25–30 percent. The work also reduced the need for “babysitting” of jobs, resulted in fewer long-running production issues and less time spent debugging legacy workflows, and enabled faster fixes and releases through the DevOps pipeline, with improved understanding through notebooks and code.

Following a successful proof of concept in which we delivered the first five applications, the bank expanded the workstream and awarded us an additional set of 20 applications for migration. As part of the collaboration, we jointly built and implemented a standardized framework to enable a consistent approach across migrations. The bank is also encouraging other vendors to follow the same framework and avoid introducing new tools.

Migration is not simply code conversion. Without a shared framework, it becomes difficult to apply standard patterns consistently, while many ancillary elements must also be addressed to run workloads end-to-end. With the framework and tooling around code generation, teams can reduce conversion time and focus more on deeper analysis and optimization.

The approach and accelerators are reusable for other organizations with Informatica-to-Databricks modernization needs.

Technologies referenced

The technologies referenced in the engagement include Databricks, Python and PySpark, along with the Databricks migration analysis tool Lake Bridge Analyzer. Azure services include Key Vault, Azure AD, Azure Data Flow, Azure Data Lake Syatem, App Insights and Vitual Machines, together with integration to an Elastic/ELK logging framework.