Title

AWS re:Invent 2023 - [LAUNCH] Achieving scale with Amazon Aurora Limitless Database (DAT344)

Summary

Amazon Aurora Limitless Database is a new scaling capability for Aurora that allows scaling beyond the limits of existing Aurora clusters.
It offers a serverless deployment of Aurora that automatically scales beyond the limits of a single instance using a distributed architecture.
Limitless Database provides transactional consistency across the entire system and supports millions of transactions per second and petabytes of data within a single Aurora cluster.
It introduces sharded tables and reference tables to distribute data across instances while maintaining the simplicity of a single database.
The system uses a distributed transaction router and data access shards to handle application traffic and scale writes as well as reads.
Limitless Database maintains Postgres compatibility, supports read committed and repeatable read isolation levels, and integrates with Aurora's distributed storage system.
It uses a custom algorithm with bounded clocks for time-based transaction consistency and supports a broad coverage of the SQL feature set.
The system is designed to optimize for single shard operations for best performance but also supports parallel operations like index creation and aggregates.
Limitless Database is available in limited preview for the Postgres compatible version of Aurora.

The Limitless Database addresses the complexity and maintenance challenges associated with traditional sharding by providing a managed service that simplifies scaling and data distribution.
The use of sharded tables and reference tables allows for efficient data organization and query execution, with the ability to co-locate related data on the same shard for optimized joins.
The system's architecture, which includes distributed transaction routers and data access shards, is designed to provide high availability and resiliency by spreading components across availability zones and offering configurable compute redundancy.
The integration of EC2 time sync service into Postgres enables Limitless Database to maintain transactional consistency across a distributed system, which is a significant technical achievement.
Limitless Database's approach to SQL compatibility and query execution leverages Postgres foreign tables and a custom foreign data wrapper, ensuring that users can continue to use familiar Postgres features and tools.
The system's design prioritizes scalability and performance, with optimizations for single shard operations and the ability to handle embarrassingly parallel tasks efficiently.
The preview availability of Limitless Database offers an opportunity for users to test and provide feedback on this new service, which could significantly impact how large-scale relational databases are managed in the cloud.