A malformed file should not hold up or back up the pipeline (resilience). This means using existing infrastructure and established patterns within the Netflix ecosystem as much as possible and minimizing the introduction of new technologies.Įqually important is the resilience, recoverability, and supportability of the solution. As with any sustainable engineering design, focusing on simplicity is very important. There are multiple ways you can solve this problem and many technologies to choose from. We found ourselves needing to hold more than 120 thousand messages in flight at a time in order to keep up with the volumes of files. The primary limitation was that AWS SQS queues have a limit of 120 thousand in-flight messages. But how does it hold up to the likes of Netflix VPC Flow Logs that has volumes which are orders of magnitude greater? It didn’t. It works well for other pipelines that have thousands of files landing in s3 per day. In other words, we are able to ensure that our Spark app does not “eat” more data than it was tuned to handle. What we get is a group of messages representing a set of s3 files which we humorously call “Mouthfuls”. In addition to the s3 object path, these events also conveniently include file size which allows us to intelligently decide how many messages to grab from the SQS queue and when to stop. As you may know, S3 can emit messages when events (such as a file creation events) occur which can be directed into an AWS SQS queue. It is easier to tune a large Spark job for a consistent volume of data. So how do we ingest all these s3 files?Īt Netflix, we have the option to use Spark as our distributed computing platform. And in order to gain visibility into these logs, we need to somehow ingest and enrich this data. With a large ecosystem at Netflix, we receive hundreds of thousands of VPC Flow Log files in S3 each hour. VPC Flow Logs are enriched using IP Metadata from Sonar as it is ingested. Sonar is an IPv4 and IPv6 address identity tracking service. To understand the attributes of each IP back to an application metadata Netflix uses Sonar. The IP addresses within the Cloud can move from one EC2 instance or Titus container to another over time.

By default, each record captures a network internet protocol (IP) traffic flow (characterized by a 5-tuple on a per network interface basis) that occurs within an aggregation interval. A flow log record represents a network flow in the VPC. Flow Logs are enabled tactically on either a VPC or subnet or network interface. At Netflix we publish the Flow Log data to Amazon S3. VPC Flow Logs is an AWS feature that captures information about the IP traffic going to and from network interfaces in a VPC. By collecting, accessing and analyzing network data from a variety of sources like VPC Flow Logs, ELB Access Logs, Custom Exporter Agents, etc, we can provide Network Insight to users through multiple data visualization techniques like Lumen, Atlas, etc. Network Availability: The expected continued growth of our ecosystem makes it difficult to understand our network bottlenecks and potential limits we may be reaching.Ĭloud Network Insight is a suite of solutions that provides both operational and analytical insight into the Cloud Network Infrastructure to address the identified problems.Without having network visibility, it’s not possible to improve our reliability, security and capacity posture. Service Segmentation: The ease of the cloud deployments has led to the organic growth of multiple AWS accounts, deployment practices, interconnection practices, etc.Pathway Validation: Netflix velocity of change within the production streaming environment can result in the inability of services to communicate with other resources.App Dependencies and Data Flow Mappings: Without understanding and having visibility into an application’s dependencies and data flows, it is difficult for both service owners and centralized teams to identify systemic issues.While we strive to keep the ecosystem simple, the inherent nature of leveraging a variety of technologies will lead us to complications and challenges such as: The Cloud Network Infrastructure that Netflix utilizes today is a large distributed ecosystem that consists of specialized functional tiers and services such as DirectConnect, VPC Peering, Transit Gateways, NAT Gateways, etc. How Netflix is able to enrich VPC Flow Logs at Hyper Scale to provide Network Insightīy Hariharan Ananthakrishnan and Angela Ho

Vpc flow logs