The Three-Stage Pipeline Behind Netflix's Service Dependency Map

Netflix runs hundreds of services. When an incident hits, the first question every engineer asks is: what depends on what? Without an answer, you are guessing blast radius, guessing root cause, and wasting minutes that matter.

Netflix's engineering team just published a deep dive into how they built Service Topology, the system that answers that question in real time. It processes flow logs, IPC metrics, and traces from multiple regions, serves queries in sub-second latency, and lets you rewind to see the dependency graph during any past incident.

The architecture is worth understanding because the problems they solved are not Netflix-specific. Every SaaS with more than a handful of services eventually needs a dependency map. Here is how they built theirs, and what went wrong along the way.

The Three-Stage Architecture

The Service Topology pipeline processes millions of events per second across three stages, each independently scalable and each designed to redistribute data before the next stage processes it.

Stage 1 is ingestion and normalization. Raw events from network flow logs and inter-process communication arrive from multiple regions. Aggregators normalize them into a common entity model: service nodes, edges between them, and properties like protocol, port, and volume. Each aggregator owns a specific entity slot and runs for a 5-minute window, producing an immutable snapshot of that entity's state for those five minutes.

Stage 2 is where scaling gets interesting. Normalized data from Stage 1 needs further aggregation, but the data is not uniform. A handful of services like the content delivery pipeline generate orders of magnitude more traffic than everything else. This power-law distribution means consistent hashing alone does not work: the hot services overwhelm whatever instance they land on.

The solution is graduated redistribution. Each downstream aggregator subscribes to data from multiple upstream aggregators. Each upstream distributes across multiple downstreams. Load spreads at every stage. Even with highly skewed data, no single instance becomes a bottleneck.

Stage 3 is persistence and enrichment. Aggregated entities get enriched with context from external sources: ownership information, deployment health, alert status. Netflix does this enrichment at aggregation time, not query time, so every topology node carries full context when queried. No post-query joins, no slow lookups.

Where Teams Mess This Up

Netflix's postmortem of their V2 pipeline reads like a checklist of distributed systems failure modes that apply at any scale.

Kafka lag. The V2 pipeline used a single Kafka topic for all intermediate data. When traffic spiked, the topic became a bottleneck, consumer lag grew, and queries returned stale topology. The fix was per-stage Kafka topics with independent scaling groups.

Hot nodes. Consistent hashing distributed data evenly until a single service generated 10x the traffic of everything else. That service's partition became a hotspot, throttling that instance while others sat idle. The three-stage pipeline with graduated redistribution solved this by ensuring no single aggregator handles all the data for a hot entity.

Memory pressure from immutability. The team initially modeled entities as immutable, replacing them on every update to capture changes. At Netflix's volume, the GC pressure from millions of short-lived allocations was catastrophic. The fix: time-windowed aggregators that batch updates into 5-minute immutable snapshots instead of per-event objects.

Backpressure cascading upstream. When downstream aggregators fell behind, backpressure propagated all the way up to the ingestion stage. Independently scalable per-stage consumer groups with backpressure-aware buffering broke the chain.

Uneven graph database throughput. Write distribution to their graph database was not even. Some partitions received heavy write traffic while others sat idle, causing throttling that limited overall throughput. Batching writes before persisting and improving distribution logic across partitions smoothed this out.

The pattern is always the same: production reveals an assumption, profiling identifies the root cause, targeted fixes improve specific metrics. There is no such thing as optimizing everything at once. You fix the current bottleneck, measure, then fix the next one. Netflix's engineers describe it as a feature, not a bug: each optimization raises throughput, which stresses the next weakest point.

The Time-Travel Feature

One of the most powerful capabilities Netflix built lets engineers query historical topology: "What did the call graph look like during the incident at 14:00 yesterday?"

The naive approach would be full snapshots (exponential storage costs) or event sourcing (slow log replay). Netflix combined two mechanisms:

Time-windowed aggregator snapshots. Every 5-minute aggregator snapshot is immutable and persists in the graph database keyed by (entity_id, timestamp). This gives checkpoint states every 5 minutes at predictable storage cost. Each snapshot is a complete state of that entity for that window.

Property-level mutation tracking. For changes between snapshots, the graph database maintains mutation history at the property level, storing only changed properties with timestamps. This is more efficient than full entity copies and provides sub-window precision beyond the 5-minute aggregation boundaries.

Query-time reconstruction combines both: retrieve the nearest snapshot, then apply mutations forward to reach the exact target time. This avoids log replay entirely and handles arbitrary time ranges without pre-computing all possibilities. They can even re-aggregate data at query time using the same aggregator classes from ingestion, enabling exploratory analysis along arbitrary dimensions like availability tier or business domain without exploding storage.

Building Your Own Dependency Map

You do not need Netflix scale to need a service dependency map. Here is the threshold: if you have ever been on-call and could not immediately tell an engineer which services would break if yours restarted, you need one.

Start small and add stages as you grow:

Stage 0: Log every call. Record every outbound HTTP/gRPC call your services make into a central log. Tag them with caller and callee service names. Use structured logging so this data is queryable from day one. Even a simple query against these logs will be faster than guessing during an incident.

Stage 1: Aggregate by entity. When raw logs grow beyond what a simple query can handle, add aggregation that groups events by (caller, callee, time_window). Store the result in a time-series database or graph database. This is where most teams can stop for a long time.

Stage 2: Isolate hot entities. When one service generates 10x the traffic of others and skews your aggregations, split the pipeline. Give hot entities their own aggregation partition. Use a multi-stage architecture where each stage redistributes before the next processes.

Stage 3: Enrich at ingest time. Once you have the core dependency graph, attach ownership, on-call, deployment, and health metadata at aggregation time, not at query time. This keeps queries fast and ensures every topology node has full context when someone needs it during an incident.

The Takeaway

Netflix's Service Topology team describes their optimization journey as "measure, hypothesize, validate, iterate." They did not get the architecture right upfront. They built V1, watched it break, fixed the most urgent bottleneck, watched the next one appear, and repeated. The three-stage pipeline with graduated redistribution and time-windowed aggregators is the result of that iterative process, not the starting point.

The lesson for builders: your dependency map does not have to be perfect on day one. But it does need to exist, and it needs to be built so each stage can scale independently. The alternative is discovering your service dependencies during an incident, which is the worst possible time to learn them.