Title

AWS re:Invent 2023 - Driving down the cost of observability (DOP224)

Summary

Chris, a former SRE and Java engineer, now a principal engineer at CoreLogix, discusses strategies to reduce observability costs.
Observability costs are often disproportionate to the value provided, sometimes due to inefficiency.
CoreLogix offers a SaaS-based observability platform with unique cost optimization tools.
The talk covers why observability is expensive, market trends, and how to use data more efficiently.
Key points include the high cost of indexing data, the repetitive nature of errors, and the short lifespan of most queries.
Chris advises against the common pattern of moving data from high-performance to low-performance storage based on age.
He recommends creating usage statistics, defining use cases, and being cautious with indexing.
Chris also touches on the importance of avoiding vendor lock-in and asking SaaS providers about their cost optimization tools.
CoreLogix's approach to cost optimization includes transparency, use case-driven data management, and leveraging open source tools like OpenTelemetry.

Observability costs can reach up to 30% of a company's cloud budget, which is considered excessive.
Microservices, cloud server proliferation, and chaos engineering contribute to increased complexity and data volume, driving up observability costs.
A significant portion of observability data is never searched or used, indicating inefficiencies in data management.
Chris emphasizes the importance of understanding data usage patterns and aligning them with specific use cases to optimize costs.
He criticizes the common practice of tiered data storage based on age and advocates for a use case-driven approach.
Open source tools and standards like OpenTelemetry are recommended to avoid vendor lock-in and facilitate easier transitions between providers.
CoreLogix's platform focuses on cost transparency and provides tools for customers to manage their data based on usage, which is not common in the industry.
The talk suggests that asking SaaS providers about specific cost optimization tools can reveal their commitment to helping customers manage costs effectively.