Title

AWS re:Invent 2022 - Best practices for managing S3 data at scale, w/Bridgewater Associates (STG337)

Summary

• Overview of Amazon S3: Amazon S3 offers virtually unlimited scale, operates in 30 regions and 96 availability zones, and is designed for 11 nines of durability. Customers use S3 for high availability, cost optimization, and security features.
• Encrypting Data at Scale: The session covered the use of S3 encryption features, including client-side encryption, encryption in transit, and encryption at rest with different key management options. It also introduced default encryption for S3 buckets and the use of S3 inventory and batch operations to encrypt existing objects.
• Archiving Workloads: The importance of archiving for cost savings, backup storage, and compliance was discussed. The session introduced S3 Glacier storage classes and the new improvements in Glacier's restore throughput. It also explained how to use S3 batch operations and event notifications for restoring data at scale.
• Querying Logs: The session explored the use of Storage Lens, inventory reports, CloudWatch metrics, server access logs, and CloudTrail data events for querying logs and understanding bucket usage.
• Managing Storage Spend: Lifecycle policies were discussed as a means to manage storage costs by transitioning and expiring data. The session also introduced new lifecycle features such as object size filtering and version count limits.
• Bridgewater Associates Use Case: Robin Anil from Bridgewater Associates shared how they use S3 to manage petabytes of data, focusing on security, reliability, and cost. They utilize features like KMS encryption, object lock, replication time control, and intelligent tiering to manage their data lake efficiently.

Insights

• Security and Compliance: The session emphasized the importance of securing data at scale, highlighting the need for encryption and compliance with organizational policies.
• Cost Optimization: The use of intelligent tiering and lifecycle policies can significantly reduce storage costs, as demonstrated by Bridgewater Associates' 35% cost reduction while their storage grew by 42%.
• Performance Improvements: The new Glacier restore throughput improvements and the ability to handle large volumes of data restores can significantly speed up access to archived data.
• Automation and Monitoring: The use of S3 batch operations, event notifications, and monitoring tools like Storage Lens and CloudWatch metrics can automate and simplify the management of large-scale data storage.
• Real-world Application: Bridgewater Associates' use case provided a practical example of how AWS S3 features are applied in a real business context, managing large-scale data with a focus on security, reliability, and cost-efficiency.
• Continuous Innovation: AWS continues to innovate with new features and improvements, such as the 34 new metrics for Storage Lens and the lifecycle enhancements, demonstrating their commitment to meeting the evolving needs of their customers.