Title

AWS re:Invent 2022 - Data pipeline automation with built-in metadata and lineage (PRT054)

Summary

The talk addresses the scaling challenges faced by data teams, with 95% at or over capacity and the shift of data from being a byproduct to being the core product of businesses.
A survey revealed that only 3.5% of data professionals currently invest in automation, but 85% plan to do so within the next 12 months, indicating a significant gap and need for automation.
The speaker discusses the shift from imperative to declarative systems in data engineering, emphasizing the benefits of automation and metadata-driven, context-aware systems.
Ascend's approach to data pipeline automation is explained, which includes a declarative control plane that operates continuously and responds to changes in code and data.
The architecture of Ascend's system is broken down into three layers: logic plane, control plane, and data plane, with a focus on the importance of metadata and the use of SHAs (secure hash algorithms) to link code and data.
Real-world applications of Ascend's platform are highlighted, including smart backfill, error recovery, incremental data propagation, and mid-pipeline data SHA computation for efficiency.
The speaker concludes by inviting attendees to visit their booth for further discussion and demonstrations.

The data industry is experiencing a significant shift towards automation due to the overwhelming capacity challenges faced by data teams.
The transition from imperative to declarative systems represents a paradigm shift in data engineering, aiming to reduce the manual effort required to manage data pipelines.
Metadata is becoming increasingly important in modern data systems, and handling it effectively is now a big data challenge in itself.
The use of SHAs to link code and data is a key innovation that allows for more efficient change detection and system integrity checks.
The benefits of a declarative control plane include continuous operation, adaptability to real-time changes, and reduced burden on developers.
The talk suggests that there is a large untapped potential for automation in data engineering, with many organizations planning to invest in it soon.
Ascend's approach to data pipeline automation could serve as a model for other organizations looking to improve their data operations through automation and metadata management.