Title

AWS re:Invent 2022 - Deploying Egress Traffic Controls in Production Environments (SEC312)

Summary

• Presenters: Graham Zuloff (Principal Solutions Architect, AWS), Houston Hopkins (Security Team, Robinhood), Kevin Park (Security Software Engineer, Robinhood).
• Egress Controls: Necessary to mitigate attacks like log4j, which require outbound communication to download payloads or connect to command and control servers.
• AWS Network Firewall: A managed service for VPC protection, scalable, high availability, and easy to set up with fine-grained access control.
• Robinhood's Journey: Adopted AWS Network Firewall for better visibility and control over egress traffic. They phased the adoption to avoid breaking production and maintain engineer satisfaction.
• Implementation Goals: Capture and monitor all egress traffic, block bad traffic immediately, and ensure the firewall does not introduce new bottlenecks.
• Deployment Models: Centralized (single firewall) vs. Distributed (firewall per environment). Robinhood chose the distributed model for scalability.
• Deployment Process: Implemented using Terraform for systematic deployment, ensuring zero downtime and easy rollback in case of issues.
• Monitoring and Alerting: Utilized CloudWatch for operational metrics, alerts for low traffic or dropped packets, and logging for visibility.
• Cost Savings: Unexpected cost savings by identifying opportunities to use VPC endpoints instead of routing through the NAT gateway.
• Further Improvements: Simplifying route tables and separating NAT gateways into their own subnets for cleaner architecture and cost savings.

Insights

• Egress Controls Importance: The talk emphasizes the critical role of egress controls in cybersecurity, particularly in preventing the exfiltration of data and blocking outbound communication from compromised systems.
• AWS Network Firewall as a Solution: AWS Network Firewall is highlighted as a robust and scalable solution for managing network traffic, with the ability to handle high volumes of connections and integrate with AWS services.
• Phased Adoption Strategy: Robinhood's approach to adopting AWS Network Firewall in phases is a best practice for minimizing disruption in production environments and ensuring that security measures do not negatively impact development workflows.
• Terraform for Deployment: The use of Terraform for deploying AWS Network Firewall showcases the benefits of infrastructure as code, such as repeatability, consistency, and the ability to quickly toggle firewall rules and routes.
• Monitoring and Alerting: The importance of monitoring and alerting is underscored, with specific examples of how CloudWatch can be used to detect potential issues with the firewall and ensure it is functioning as expected.
• Cost Management: The session provides an interesting insight into how security tools like AWS Network Firewall can lead to cost savings by revealing inefficient traffic patterns and enabling better use of AWS services like VPC endpoints.
• Continuous Improvement: The talk concludes with a discussion on further architectural improvements, demonstrating a commitment to continuous refinement and optimization of security controls.