Title

AWS re:Invent 2022 - Publishing real-time financial data feeds using Kafka (ARC314)

Summary

• Speakers: Rana Dutt (Principal Solutions Architect) and Diego Suarez (Senior Solutions Architect), both specializing in financial services at AWS.
• Topics Covered:
  • Types of financial data streams: stock quotes, corporate actions, analytics embedded within data streams.
  • Use cases: trading, surveillance and fraud detection, risk assessment, post-trade analytics, and backtesting.
  • Network latency considerations for trading applications on AWS vs. proximity to exchanges.
  • Traditional market data distribution vs. AWS Local Zones for low latency requirements.
  • Amazon Managed Streaming for Kafka (MSK) as a service to deploy Kafka on AWS.
  • Architecture for publishing financial data feeds on AWS using MSK.
  • Monetization of data feeds through AWS Marketplace.
  • Security model for authenticating and authorizing clients using Kafka.
  • Operational aspects: monitoring and automatic deployment.
• Key Points:
  • Kafka's popularity in financial services due to high throughput, low latency, and low jitter.
  • MSK simplifies Kafka deployment and management, offering high availability and scalability.
  • Public and private connectivity options for clients to access Kafka clusters.
  • Use of AWS PrivateLink for secure, private data streaming as a service.
  • Integration with AWS Marketplace for monetization.
  • Security through TLS, digital certificates, and Kafka ACLs.
  • Monitoring with CloudWatch and Prometheus, and logging with CloudWatch, Amazon S3, and Amazon Kinesis.

Insights

• Kafka's Suitability for Financial Data: Kafka's architecture, with its publish-subscribe model and partitioning, is particularly well-suited for financial data streams, which require efficient and reliable real-time data delivery.
• MSK as a Managed Service: MSK addresses the operational complexity of managing Kafka clusters, allowing financial institutions to focus on their core business logic and data, rather than infrastructure management.
• Hybrid Architectures for Latency: AWS Local Zones can be used to meet low latency requirements for trading applications, offering an alternative to co-location at exchange data centers.
• Monetization Strategy: The integration of AWS Marketplace provides a streamlined path for financial data providers to monetize their feeds, leveraging AWS's extensive customer base and simplified billing and subscription management.
• Security and Compliance: The detailed explanation of the security model, including TLS, digital certificates, and Kafka ACLs, highlights AWS's commitment to providing secure and compliant solutions for sensitive financial data.
• Operational Excellence: The emphasis on monitoring and logging demonstrates the importance of operational excellence in managing financial data feeds, ensuring reliability and performance for end-users.
• AWS PrivateLink: The use of AWS PrivateLink for secure data streaming as a service is a significant advantage for SaaS providers, as it simplifies network architecture and enhances security by keeping traffic within the AWS network.