Title

AWS re:Invent 2023 - [LAUNCH] Deep dive on Amazon S3 Express One Zone storage class (STG230)

Summary

• Amazon S3 Express One Zone is a new high-performance storage class designed for low latency and high throughput.
• It offers up to 10 times faster access speed than Amazon S3 standard and supports millions of requests per minute.
• The storage class uses a single zone architecture on purpose-built hardware, introduces S3 directory buckets, and a new session-based authorization model.
• S3 Express One Zone is integrated with key AWS services like CloudWatch, CloudTrail, CloudFormation, IAM, VPC, EMR, Athena, SageMaker, EKS, EC2, Lambda, and developer tools.
• It is also compatible with AWS partner services for analytics, machine learning, media editing, governance, and backup.
• The storage class is ideal for use cases requiring high performance, such as interactive analytics, video editing, and machine learning pipelines.
• Pinterest shared their experience using S3 Express One Zone to accelerate their machine learning pipelines and achieve significant improvements in workload speed and cost savings.
• The session included a demo showing how to create an S3 directory bucket, import data, and perform performance tests comparing S3 Express One Zone to S3 standard.

Insights

• S3 Express One Zone's single zone architecture means data is stored in one AZ, which can reduce latency but also introduces a risk of data loss if the AZ is compromised.
• The new S3 directory buckets are designed to handle hundreds of thousands of transactions per second, allowing for quick scaling and high transaction rates.
• The session-based authorization model, S3 Create Session, optimizes latency by providing temporary credentials for low-latency access to the bucket.
• S3 Express One Zone can significantly reduce job or query time for latency-sensitive applications, as demonstrated by Ansatz Capital's 78% improvement in workload speed.
• The storage class can also lead to cost savings, with customers potentially saving up to 60% in total cost of ownership due to reduced compute idle time and lower request costs.
• AWS services like Athena and EMR can run up to 2x and 4x faster, respectively, when using S3 Express One Zone as a caching layer or intermediate data store.
• Amazon SageMaker machine learning applications can run 5.8 times faster with S3 Express One Zone compared to S3 standard in fast file mode.
• The integration with AWS partner services allows customers to leverage the performance benefits of S3 Express One Zone across a wide range of applications and industries.
• Pinterest's use case demonstrates the practical application of S3 Express One Zone in a real-world scenario, highlighting the benefits of improved performance and cost efficiency for large-scale data processing workloads.