Title

AWS re:Invent 2022 - Observability that scales: Analyze data in stream and query remotely (PRT233)

Summary

• Observability costs account for 7-10% of cloud costs, with storage being a significant expense.
• The rise of open telemetry and structured data has improved integration and consistency.
• Infinite storage and tiered options are available, but data volume and distributed systems present challenges.
• CoreLogix's approach focuses on analyzing data in-stream and storing only what matters in a remote archive.
• CoreLogix's architecture is based on Kafka streams and processes data without leaving the stream, using a "store, stream, and sync" mechanism.
• Nutanix leverages CoreLogix for SaaS application monitoring, achieving full-stack observability with lower TCO.
• CoreLogix's features include log aggregation, flow alerts, and Data Prime, a pipeline search syntax for complex queries.

Insights

• Observability is a significant part of cloud costs, and optimizing storage is crucial for managing expenses.
• The adoption of open telemetry and structured data has made it easier to integrate and manage observability data.
• Despite the availability of infinite storage, the sheer volume of data and the need for quick access to recent data make it challenging to manage costs effectively.
• CoreLogix's in-stream analysis and remote querying approach aim to address the challenges of data volume and cost by storing only essential data and using cost-effective storage solutions like Amazon S3.
• Nutanix's use case demonstrates the practical benefits of CoreLogix's approach, including cost savings, improved observability, and the ability to avoid vendor lock-in.
• CoreLogix's architecture and features like flow alerts and Data Prime suggest a trend towards more sophisticated and efficient observability tools that can handle large volumes of data and complex queries without compromising on performance or cost.