Title

AWS re:Invent 2023 - Operating with AWS Open Source Observability (COP332)

Summary

• AWS customers are increasingly interested in open source software for observability due to its flexibility, integration capabilities, and vendor neutrality.
• Operating the infrastructure for open source software can be challenging, but AWS Managed Open Source Observability Services offer a solution.
• Imayakumar Jagannathan, a Principal Solution Architect at AWS, discusses real-world use cases of customers facing observability challenges and how they leveraged open source software.
• Gustavo Franco from Vitex shares their journey with AWS-managed open-source services for observability needs.
• Marc Chenet provides insights into operating these services at scale and introduces new features and launches.
• Observability is defined as the ability to infer the internal state of a system by looking at external signals, with monitoring being the actions performed to achieve this.
• AWS offers both native observability services (CloudWatch, X-Ray, etc.) and open source managed services (Amazon Managed Grafana, Amazon Managed Service for Prometheus, Amazon OpenSearch Service).
• AWS Distro for OpenTelemetry supports any source to any destination, processing signals in memory, and is a redistribution of the upstream OpenTelemetry project with AWS support.
• Amazon Managed Service for Prometheus is a serverless solution for metric monitoring, offering high availability and scalability without infrastructure concerns.
• Amazon OpenSearch Service is a managed service for log analytics and real-time application monitoring.
• Amazon Managed Grafana provides a managed Grafana environment with enterprise-grade plugins and data sources.
• Customer use cases include Fidelity's migration to AWS and adoption of open source observability, Philips 66's transition from legacy monitoring to a fully open-source stack, Choice Hotels' modern use cases with tracing and metrics extraction, and Northwestern Mutual's migration from self-hosted Prometheus to a managed environment.
• Vitex's observability journey involved addressing challenges such as budget overrun, lack of policy control, vendor sprawl, and improving developer experience.
• AWS's internal use of open source observability services includes monitoring and managing costs, ensuring high availability, and contributing to open source projects.
• New features for AWS observability services include agentless collectors for EKS, support for logs in AWS Distro for OpenTelemetry, 300+ community plugins for Amazon Managed Grafana, and more.

Insights

• The growing trend among AWS customers towards open source observability tools is driven by the desire for customization, integration, and avoiding vendor lock-in.
• AWS's managed services for open source tools aim to alleviate the operational burden associated with running these tools at scale.
• Real-world customer stories, such as those from Fidelity, Philips 66, Choice Hotels, and Northwestern Mutual, illustrate the diverse needs and benefits of adopting AWS's managed open source observability services.
• Vitex's observability journey highlights the importance of governance, cost control, and developer experience in observability practices.
• AWS's internal practices for monitoring their managed services, such as using cell-based architecture and monitoring monitors, reflect a commitment to operational excellence and can serve as a model for other organizations.
• The introduction of agentless collectors and support for logs in AWS Distro for OpenTelemetry indicates AWS's continuous innovation in simplifying observability for their customers.
• The session underscores the importance of community involvement and open source contributions, which AWS actively supports and encourages among its engineers and customers.