AWS re:Invent 2023
Data Processing at Massive Scale on Amazon Eks Con309

Title

AWS re:Invent 2023 - Data processing at massive scale on Amazon EKS (CON309)

Summary

  • Speakers: Alex Lines (AWS Senior Container Specialist), Vara Bontu (AWS Principal Solutions Architect), and Soam Acharya (from Pinterest).
  • Topics Covered:
    • Kubernetes for Data Processing: Scalability, orchestration, portability, and standardization are key reasons for adopting Kubernetes for data workloads.
    • Data on EKS Project: AWS's initiative to provide blueprints for building modern data platforms on EKS, including reference architectures, IAC templates, and sample code.
    • Open Source Data Platforms on EKS: Discussion on running Apache Spark and other data processing tools on Kubernetes, with a focus on Spark's integration as a resource manager.
    • Pinterest's Modernization Journey: Transition from Hadoop to Spark on EKS, including the challenges and benefits encountered.
    • Best Practices for Running Apache Spark at Scale: Addressing common challenges such as compute-intensive workloads, volatile scaling patterns, and high availability.
    • Blueprints for Machine Learning and Data Processing: AWS's focus on providing blueprints for prevalent use cases among customers.
    • Technical Deep Dive: Detailed discussion on networking configurations, storage options, compute considerations, auto-scaling, batch scheduling, metrics, logging, and best practices for running Spark on EKS.

Insights

  • Adoption of Kubernetes for Data Workloads: The move towards Kubernetes for data processing is driven by the need for scalability, fine-grained resource control, and the ability to run containerized applications in a standardized environment.
  • Data on EKS Project: AWS's commitment to helping customers build data workloads on EKS is evident through the Data on EKS project, which aims to simplify the deployment process and ensure adherence to AWS best practices.
  • Open Source Data Platforms: The integration of open source data platforms like Apache Spark with Kubernetes is becoming more seamless, with Kubernetes now supported as a resource manager for Spark.
  • Pinterest's Experience: Pinterest's journey to modernize their data processing by moving from Hadoop to Spark on EKS highlights the practical considerations and benefits of such a transition, including cost savings and improved agility.
  • Best Practices for Scalability: The session provided valuable insights into best practices for running Apache Spark at scale on EKS, addressing common issues such as IP exhaustion, DNS resolution, storage performance, and compute optimization.
  • Community Engagement: AWS encourages community engagement and contributions to their open source projects, emphasizing the collaborative nature of the Kubernetes and data processing ecosystems.
  • Future Directions: The talk suggests a trend towards more organizations migrating their data workloads to Kubernetes-based platforms like EKS, with a focus on leveraging the benefits of cloud-native solutions and open source tools.
Data Patterns Get the Big Picture for Data Applications Pex309Data Protection and Resilience with Aws Storage Stg215 R