Title

AWS re:Invent 2023 - BMO’s 1.5 PB migration & modernization journey with AWS ProServe (PRO302)

Summary

BMO, after acquiring Bank of the West, embarked on a journey to migrate 1.8 petabytes of data to the cloud with AWS ProServe.
The migration involved data warehouses and data marts, with a focus on small steps that set the tone for the entire process.
The team faced challenges such as extracting from live systems, dealing with large volumes and complexity, and slow network speeds.
They used AWS Snowball for offline data transfer due to the slow network, which would have taken over three years to transfer the data online.
The migration required careful planning, including prioritizing data, automating processes, and setting ambitious goals like 24/7 extraction and ingesting 100 terabytes of data per day.
They implemented a lake house architecture using AWS services like SCT, Glue, Redshift, and DataSync, and built a self-service, user-friendly data access provisioning process.
The team created repeatable frameworks and automation for data extraction, ingestion, and validation, ensuring data security and governance.
The project delivered more capabilities and patterns for future cloud migrations, accelerating the delivery of actionable insights to the business.

The migration was a post-acquisition effort, which introduced additional timelines and technical constraints.
The use of AWS Snowball was critical due to the initial network limitations, highlighting the importance of offline data transfer options in large migrations.
The project emphasized the importance of automation and the creation of repeatable frameworks to handle the scale of data and complexity.
Data security and governance were top priorities, with encryption of data in transit, at rest, and additional encryption for sensitive data.
The team faced AWS service limits and had to work closely with AWS support to overcome these challenges.
The migration not only moved the data but also modernized the data access provisioning process, improving user experience and management.
The project's success has laid the foundation for future cloud migrations at BMO, showcasing the scalability and reusability of the developed frameworks and automation patterns.
The session highlighted the collaborative effort between BMO, Bank of the West, and AWS ProServe, demonstrating the importance of partnership in successful cloud migrations.