Title

AWS re:Invent 2022 - Virtually unlimited scale and hybrid SaaS deployments with Matillion (PRT106)

Summary

• Presenters: Molly Sanbo, Director of Product Marketing, and Tom Ridings, Director of Product Management at Matillion.
• Context: Businesses are increasingly moving data to the cloud, creating a demand for efficient data transformation.
• Matillion's Solution: Matillion's Data Productivity Cloud helps teams connect, transform, and synchronize data, improving value realization.
• Push-Down Transformation: Matillion uses a push-down approach, generating SQL to execute transformation jobs directly in the cloud data warehouse, avoiding the need for dedicated ETL servers.
• Benefits of Push-Down:
  • No data movement in and out of the warehouse, reducing networking costs and performance impacts.
  • Transformations are expressed in SQL, which is more accessible to data teams.
  • Utilizes the scaling capabilities of the warehouse without additional overhead.
• Unlimited Scale Feature:
  • Introduces automatic scaling for large workloads, scaling up and down as needed without manual intervention.
  • More granular consumption of Matillion credits, paying only for what is used.
  • Data remains within the user's cloud, maintaining security and sovereignty.
• ETL Agents: Serverless design allows for parallelism and backward compatibility with existing jobs.
• Hybrid SaaS Deployment: Combines the benefits of SaaS with self-hosting, managing jobs from the Matillion Hub while running ETL agents in the user's cloud.
• Live Demo: Showcased a scenario of a holiday letting review website with multiple databases, demonstrating how unlimited scale handles concurrent jobs and infrastructure scaling.
• Additional Features: Matillion Data Productivity Cloud includes standardized metadata, centralized observability, and supports various workloads.
• Availability: Unlimited Scale is in private preview, with an invitation for attendees to visit the Matillion booth for more information.

Insights

• Shift to Cloud Data Warehousing: The transition to modern cloud warehouses has allowed for a rethinking of data warehouse design, with a focus on on-demand compute and storage.
• SQL as a Transformation Language: The push-down approach emphasizes the use of SQL for transformations, which could indicate a trend towards simplifying data pipeline complexity and making it more accessible to a broader range of users.
• Automatic Scaling: The introduction of automatic scaling is a significant advancement, as it addresses the challenge of managing fluctuating workloads and infrastructure needs in real-time.
• Hybrid Deployment Model: The hybrid SaaS model presented by Matillion suggests a growing need for solutions that offer the flexibility of cloud services while retaining control over data processing and location.
• Serverless ETL Agents: The use of serverless ETL agents reflects the industry's move towards serverless architectures, which can provide cost savings and operational efficiencies.
• Focus on Data Sovereignty: The emphasis on data never leaving the user's cloud highlights the increasing importance of data sovereignty and security in cloud-based data management solutions.
• End-to-End Data Management: Matillion's suite of features suggests a market demand for comprehensive data management platforms that can handle the entire data lifecycle from ingestion to transformation and orchestration.