Meta Rewrote Storage for AI. GPU Stalls Are the Reason.

GPU compute performance roughly triples every two years. Storage and interconnect performance do not. That gap is the single biggest source of GPU stalls in AI training, and it is getting worse as models grow faster than the hardware that feeds them.

Meta published a detailed breakdown of how its BLOB-storage architecture evolved to solve this problem. The post is worth reading in full -- it is the rare engineering write-up that names the actual trade-offs, not just the wins. Here is what they did and why it matters beyond Menlo Park.

The Bottleneck Nobody Talks About

A training run at Meta's scale involves hundreds of thousands of GPUs iterating over petabytes of data across multiple epochs. Every few steps, all those GPUs synchronize state. If one GPU is late because it is waiting on storage, every other GPU in that ring waits too.

The problem is not average latency. It is tail latency at the pMax level. Legacy BLOB storage was built for Facebook photos and Instagram reels -- workloads where a 300ms metadata lookup is fine. AI training is the opposite: bursty, sustained high throughput, predictable latency requirements, and zero tolerance for stragglers.

Meta's numbers tell the story. Their old getObject API could take hundreds of milliseconds for a single request because the metadata path went through three separate layers -- namelayer, volumeslayer, containerlayer -- each with its own stateful metadata store. A slow response from any layer cascaded into a GPU stall that multiplied across the entire training cluster.

What They Tore Down

Meta's storage is built on Tectonic, a horizontally scalable block storage layer that has been running for years. On top of Tectonic sit the BLOB-storage layers that serve Facebook, Instagram, Reality Labs, Meta AI, Ads, and the data warehouse. That architecture evolved organically, adding service-oriented layers as new products appeared.

When they benchmarked it against AI workloads, the design assumptions that made sense for social apps fell apart:

• Performance and latency. The legacy stack assumed modest latency requirements were fine. AI needs predictable bounded latency all the way to pMax.
• Reliability vs availability. Data and metadata were globally replicated by default. AI workloads need high availability but do not need global-by-default replication.
• Cost efficiency. The legacy stack optimized for cost-per-byte on HDDs. AI needs IOPS from flash, and the computational cost of storage is negligible compared to GPU cost anyway.
• Power efficiency. Data centers are now power-constrained, not space-constrained. Every kilowatt spent on storage is a kilowatt not spent on GPUs.

That last point is worth sitting with. In 2026, the storage team's design constraints include a power budget directly traded against compute. This is the new reality of AI infrastructure: you optimize for watts per IOPS, not dollars per gigabyte.

The Rebuild: Three Bets

Meta made three foundational changes.

Unified metadata schema. They collapsed the multi-layer metadata path into a single flat schema backed by ZippyDB. A getObject call that used to traverse namelayer, volumeslayer, and containerlayer now resolves in O(1) per chunk. This is a step-function improvement -- from hundreds of milliseconds to single-digit milliseconds for metadata lookups.

No dataplane proxy. The old architecture proxied all data through a middle layer between the client and Tectonic storage servers. They ripped that out and built a fat client SDK with Tectonic's BlockClient embedded directly. The client now streams bytes straight from storage servers. This eliminates the proxy's power draw and removes a throughput bottleneck.

Regional deployment. The new BLOB-storage stack is lean enough to deploy as a regional service colocated with GPUs in every AI region. Instead of one global storage fabric, each AI region gets its own storage stack. Data that needs to cross regions does so through the tiered caching system, not through synchronous metadata lookups.

Handling the Spikes

Even with a fast metadata path, AI workloads create traffic patterns that can melt any storage system. Checkpoint events trigger sharp egress spikes across hundreds of GPUs simultaneously. Model weights are perpetually hot. GPU restarts cause burst re-reads.

Meta solves this with two caching layers:

• Distributed data cache. Spare memory on GPU hosts is pooled into a distributed cache using components from Meta's existing Owl subsystem. The cache is baked into the BLOB-storage client SDK, so every read hits the cache first. Average hit rate: 80%.
• Readplan metadata cache. The mapping from path to storage address (the readplan) is cached in a distributed memory store similar to memcache. Access time: 1-2ms.

These two caches absorb spikes, eliminate metadata hot shards, and improve p50 and p99 latencies by serving reads from memory instead of hitting storage.

The Tail Fixes

The last 20% of the problem came from classic distributed systems sharp edges:

• Laggards. One slow storage node wrecking tail latency. Meta's fix: hedged reads on the client side. Send the same read request to multiple nodes and take the first response.
• Egress spikes. Checkpoint events causing congestion, timeouts, and retries. Fix: dynamic concurrency control in the client SDK that tunes parallelism based on application-level congestion signals.

Research Velocity: The Tiered Cache Model

The GPU utilization problem was only half the story. The other half is research velocity.

Researchers at Meta go through a slow loop: curate data, pick a region, submit a data ingestion job, wait hours for the snapshot to copy, then start training. If the model needs tweaking, the whole loop starts again. At a time when new frontier models ship in weeks, waiting hours per iteration is unacceptable.

Meta's insight: treat storage like a planet-scale virtual disk with tiered caching, borrowed from operating system design. A Linux process reading a file does not manually copy data between cache levels. The OS just hydrates on demand from L1 cache to L2 to main memory. Apply the same principle to datacenter storage.

• L1 cache: GPU host memory
• L2 cache: GPU host flash (NVMe)
• L3 cache: Regional disaggregated flash pool, backed by the global BLOB fabric on HDDs as the source of truth

The BLOB-storage SDK now exposes a prefetch() API. Dataloaders call it in the background to hydrate data from remote storage onto the local region's L3 cache before training needs it. The prefetch also prewarms the metadata cache, so path-to-address lookups are instant when training actually starts.

Data in the L3 cache follows TTL or LRU eviction policies, capacity-aware. Epoch reuse across a training cycle happens naturally because the cache holds data for the configured window.

The result: ingestion times dropped from hours to minutes. That is the difference between shipping a model in weeks versus months.

What This Means for Operators

You are not Meta. You do not have hundreds of exabytes of storage or a team that can rebuild your storage stack from scratch. But the design principles apply at any scale.

• Metadata path length is latency. Every hop between storage layers adds tail latency. If your object store requires three lookups to resolve a path, you are burning GPU cycles. Measure your p99 metadata latency before you buy more GPUs.
• Cache everything close to compute. Meta gets 80% hit rate from spare GPU host memory. You can get similar ratios from local NVMe on your training nodes. Do not send every read to network storage.
• Power efficiency is a storage design constraint now. If your data center is power-constrained, the storage stack that draws less power per IOPS wins. This means flash over HDD for active datasets, and no dataplane proxies that burn watts for no throughput gain.
• Hedged reads are cheap insurance. If your storage layer has tail latency problems, hedged reads on the client side are a single weekend project that can cut p99 by 10x.
• Pretend your data is a tiered cache. The operator mindset of copying data to where the compute is works for large long-running jobs but kills iteration speed for the 90% of smaller jobs that make up most AI research. A transparent tiered cache with prefetch lets researchers iterate in minutes, not hours.

The Interaction with Compute Evolution

This storage architecture shift coincides with AWS launching Graviton5-based C9g instances, which deliver DDR5 8800MT/s memory and 5x larger L3 caches than Graviton4. Faster memory and larger caches directly address the same problem Meta is solving at the storage layer: reducing data access latency. The industry is attacking the GPU starvation problem from both sides -- faster compute memory on one side, smarter storage architecture on the other. If you are building AI infrastructure today, you need to optimize both ends of that pipe.