Title

AWS re:Invent 2023 - Stripe: Architecting for Observability at Massive Scale (FSI319)

Summary

Importance of Observability: Observability is crucial for businesses to quickly respond to system failures and gain valuable insights for informed decision-making.
Challenges of Scale: As businesses grow, the complexity and cost of managing observability data increase due to factors like microservices, dynamic scaling, and distributed systems.
AWS Observability Services: AWS offers a range of services like CloudWatch, X-Ray, Amazon Managed Service for Prometheus, and Amazon Managed Grafana to help architect scalable observability solutions.
Stripe's Observability Architecture: Stripe faced challenges with scale, reliability, and cost. They implemented five architectural changes: sharding, aggregation, tiered storage, streaming alerts, and isolation.
Cultural Shifts: Stripe emphasized the need for a culture of self-reliance within the observability team and making it easy for users to do the right thing with observability practices.

Limited Unique Queries: Despite a large number of alerts, there are only a few dozen unique queries used in most alerts, suggesting that users prefer declarative alerts over complex query languages.
Data Usage: A significant portion of observability data (80-98%) is never referenced, indicating potential cost savings if unused data can be identified and managed differently.
Trade-offs in Observability: The trade-offs for observability data are different from other systems, with a preference for speed and cost-effectiveness over absolute accuracy.
Sharding Early: Sharding observability databases early can help manage scalability and user experience as the business grows.
Aggregation and Tiered Storage: Aggregation can reduce the volume of data sent to the time series database, while tiered storage can maintain optionality of data at a lower cost.
Streaming Alerts: Decoupling alerting from the time series database allows for high-cardinality metrics to be processed in memory, offering flexibility and cost savings.
Isolation for Reliability: Observability systems should be isolated from other company technologies to minimize the risk of simultaneous outages.
Cultural Importance: Cultivating a culture of self-reliance and making it easy for users to adopt proper observability practices is as important as the technical architecture.