Title

AWS re:Invent 2023 - What’s new with AWS data integration (ANT220)

Summary

Santosh Chandrachud, GM for AWS Data Integration, Sean Myron, GM for MWA and orchestration, and Nishit Desai, MD at Goldman Sachs, presented the session.
The session covered the importance of data integration, AWS's approach to it, key investments, and new announcements.
Customer success stories were shared, including BMW, Itau Bank, Merck, Chime Financial, BMO, JPMC, BMS, and GoDaddy.
AWS Glue supports hundreds of thousands of customers and millions of data integration jobs monthly.
AWS's data integration process is conceptualized into four core pillars: Connect, Transform, Operationalize, and Manage Data Quality.
New announcements included support for third-party data warehouses and databases, enhancements to first-party connectors, built-in transformations, DBT trusted adapter, large worker types, streaming improvements, job authoring time reduction, Generative AI capability with Amazon Q, and ML capabilities for data quality.
Sean Myron discussed Apache Airflow and Amazon Managed Workflows for Apache Airflow (MWAA), highlighting new features and capabilities.
Nishit Desai shared Goldman Sachs' journey with AWS Glue, emphasizing the business impact and key wins from their data integration strategy.

AWS Glue is a central component of AWS's data integration strategy, with a focus on serverless, scalable, and open-source compatible engines.
The introduction of third-party connectors and enhancements to first-party connectors like Redshift and OpenSearch indicates AWS's commitment to interoperability and multi-cloud strategies.
AWS is investing in simplifying the data integration process for users of all technical levels, from data scientists to business analysts, by providing various tools and interfaces.
The launch of Generative AI capabilities with Amazon Q for data integration suggests AWS is exploring the intersection of AI and data management, potentially revolutionizing how data pipelines are authored and managed.
The case study from Goldman Sachs demonstrates the real-world impact of AWS data integration tools, showcasing significant improvements in workflow completion time, data growth support, and system availability.
AWS's rapid feature release cadence, with over 70 new features in 11 months, reflects the company's aggressive innovation strategy to stay ahead in the data integration market.
The session highlighted the importance of data quality management, with AWS introducing machine learning capabilities to detect anomalies and improve data quality, which is critical for businesses to make informed decisions.