Title

AWS re:Invent 2022 - FINRA: Democratize data and solve issues faster with Grafana (COP210)

Summary

• Introduction: Helen Lin from AWS and FINRA representatives Sumilata Bhacchus and Logan Kalachick discuss the importance of data-driven decisions and the challenges of accessing and visualizing data.
• Observability: The concept of observability is explained as leveraging metrics, logs, and traces to monitor application and infrastructure performance, alert on SLA breaches, and enable teams to build resilient applications.
• Data Democracy: The session covers the idea of data democracy, which involves sharing data and tools across an organization to enable data-driven decisions. Challenges include data silos, security, governance, scalability, and visualization.
• AWS Solutions: Amazon Managed Grafana and Amazon Managed Service for Prometheus are presented as solutions to consolidate operational and business data, offering scalability, security, and compliance.
• FINRA Use Cases: Sumilata Bhacchus shares two use cases from FINRA:
  1. Data democratization for consolidated remote workspace tools, using AWS Data Lake and Amazon Managed Grafana to create executive dashboards for decision-making.
  2. Observability for EKS workloads, using Amazon Managed Prometheus and Managed Grafana to monitor cluster health and application performance.
• Demos: Logan Kalachick demonstrates how FINRA uses Managed Grafana to visualize data for remote work tools and EKS workloads, highlighting features like time filters, legends, toggles, and templatized dashboards.
• Future Possibilities: FINRA plans to use the collected data for hybrid work analysis, cloud service portfolio trends, application reliability KPIs, user engagement metrics, investment strategy, and software deployment efficiencies.
• Conclusion: The session concludes with a call to action to use the provided resources (observability workshop and GitHub project) and an invitation to take surveys and engage with the presenters.

Insights

• Data-Driven Culture: Both AWS and FINRA emphasize the importance of a data-driven culture, where decisions are made based on data rather than intuition or guesswork.
• Challenges in Data Access and Visualization: The common challenges faced by organizations include accessing the right data at the right time, data silos, security and governance concerns, and the need for effective visualization tools.
• AWS as an Enabler: AWS services like Managed Grafana and Managed Prometheus are positioned as enablers for overcoming these challenges by providing scalable, secure, and compliant platforms for data visualization and observability.
• Practical Application: FINRA's use cases demonstrate the practical application of AWS services in a real-world regulatory environment, showcasing how they can be used to improve operational efficiency and decision-making.
• Importance of Observability: The session underscores the critical role of observability in maintaining application resiliency and providing real-time visibility into infrastructure and application layers.
• Templatized Dashboards: The use of templatized dashboards in Grafana is highlighted as a key feature that saves time and effort when creating and deploying dashboards across different environments or for different teams.
• Future Focus: FINRA's future focus on hybrid work analysis, cloud service trends, and user engagement metrics indicates a forward-thinking approach to leveraging data for continuous improvement and strategic planning.
• Resources for Implementation: The session provides resources such as an observability workshop and a GitHub project to help attendees implement similar solutions in their own environments.