Title

AWS re:Invent 2023 - How Electronic Arts Modernized Its Data Platform with Amazon EMR (ANT320)

Summary

• Electronic Arts (EA) modernized its data platform to handle increasing data volumes and complexity using Amazon EMR.
• Arturo Bayo, AWS Data and Analytics Specialist, introduced the session alongside Shivika Verma, Amazon EMR Principal Product Manager, and Alex Ignatius, EA's Senior Director for Data and Analytics Engineering.
• The motivation for modernizing data platforms is the rapid growth of global data, which necessitates scalable solutions.
• Amazon EMR is a cloud big data solution that supports petabyte-scale data processing and machine learning with open-source frameworks.
• EMR offers cost savings, ease of use, scalability, and security, with various deployment models including EC2, EKS, serverless, and AWS Outposts.
• EA's legacy data platform faced challenges such as aging technology, ineffective auto-scaling, unpredictable SLAs, and high operational overhead.
• EA's modernization with EMR involved a seamless transition, handling mixed workloads, optimizing DevOps, and providing predictable SLAs.
• The migration strategy included job classification, t-shirt sizing for clusters, zero downtime, and observability throughout the process.
• Post-migration, EA saw significant improvements in data processing SLAs, cost savings, and the ability to handle unexpected data volume increases.
• The session concluded with lessons learned, emphasizing the importance of understanding migration benefits, deconstructing environments into workloads, building a migration plan, considering post-migration optimizations, and evolving the platform as the business grows.

Insights

• The exponential growth of data is a driving force behind the need for modern data platforms that can scale efficiently.
• Amazon EMR's decoupling of compute and storage allows for independent scaling and cost optimization, which is crucial for large-scale data processing.
• EA's approach to modernization focused on minimizing disruption to ongoing operations, which is a common concern for enterprises with continuous service requirements.
• The use of t-shirt sizing for clusters is an innovative approach to standardize resource allocation and simplify decision-making for engineers during migration.
• EA's partnership with AWS Professional Services highlights the value of leveraging external expertise for complex migration projects.
• The observability stack, including CloudWatch, Prometheus, and Grafana, played a critical role in ensuring the migration's success by providing detailed monitoring and alerting.
• The session's lessons learned provide a valuable framework for other organizations considering similar migrations, emphasizing strategic planning, workload analysis, and continuous platform evolution.