Meta Replaced the Kernel Scheduler for Ads

The Core Idea

Every millisecond of latency on an ad request costs money. Not in theory, in actual revenue. At Meta's scale, 5 million requests per second hit the serving platform entry point. That is 400 billion requests a day. A few milliseconds of p99 degradation means fewer ads ranked, worse relevance, and weaker ROI for advertisers.

When a Linux kernel upgrade threatened exactly that, Meta did not just revert the upgrade. They wrote their own CPU scheduler.

What Happened

Meta's ad serving fleet ran on Linux kernel 6.4 with the Completely Fair Scheduler (CFS). CFS had been the default Linux scheduler for years and was well understood. When they started upgrading to kernel 6.9, the new Earliest Eligible Virtual Deadline First (EEVDF) scheduler (introduced in kernel 6.6) caused a measurable latency regression. The number of ads ranked per request dropped.

EEVDF is not a bad scheduler. It improves fairness and interactivity for desktop and general server workloads. But fairness is not what ad serving needs. Ad serving needs one thing: get the latency-critical request thread on a CPU as fast as possible, and keep it there.

The standard response would be: stay on the old kernel. Some hosts did exactly that, creating technical debt and operational fragmentation. But a subset of the fleet could not stay on 6.4 forever. Security patches, hardware enablement, and new kernel features would eventually force the upgrade.

Enter sched_ext, a BPF-based extensible scheduling framework that had just been upstreamed into kernel 6.12. Developed by Meta in partnership with Google's ghOS team, sched_ext lets you write a scheduling policy as a BPF program that runs in user space. No kernel fork. No custom patch set. No months-long validation cycle.

How It Works

The scheduling policy soft-partitions CPUs into two pools: one for threads on the latency-critical request path, and one for everything else. Which thread goes where is domain-specific knowledge encoded directly in the BPF program. The pool sizes adjust dynamically based on load heuristics.

By keeping related work on the same CPUs, the policy improves L3 cache locality and reduces costly DRAM access. That is the kind of optimization a general-purpose scheduler cannot do. CFS and EEVDF do not know which threads matter more. The BPF scheduler does, because it was written by the team that understands the workload.

The policy itself is packaged as a user-space binary that loads the BPF program. To roll out a change, you restart the scheduler process. No kernel rebuild. No reboot. This is the architectural insight that makes sched_ext transformative: the scheduling policy is now a deployable artifact, not a kernel patch.

The Numbers

The initial launch switched the largest ad serving server type from kernel 6.4 with CFS to kernel 6.9 with sched_ext. The results:

• 28% reduction in ads retrieval stage tail (p99) latency
• 3.28 megawatts of power savings
• 1.1% increase in the number of ads ranked

Then the compounding improvements began. Two follow-on scheduler policy updates, shipped as purely user-space changes, delivered:

• Additional 60% reduction in service p99 latency
• 18% reduction in timeout errors on the critical path

Each iteration shipped in days, not months. The scheduler policy became a continuous optimization platform, not a one-time kernel patch.

The power savings deserve attention. 3.28 MW is not a side effect. When you schedule work more efficiently, CPUs spend less time context-switching and cache-missing. They finish work faster and can enter lower power states sooner. A power saving of 3.28 MW at a single server type is the kind of number that changes infrastructure planning.

What This Means for Product Builders

Three lessons here that apply well beyond Meta.

The cost of basic acceptability. General-purpose infrastructure is designed to be acceptable for everyone, optimal for nobody. When your latency SLA directly maps to revenue, the default scheduler is leaving money on the table. Meta quantified this: 1.1% more ads ranked. At their scale, that is a lot of money. At your scale, the question is whether your workload has similar untapped headroom.

BPF changes the upgrade calculus. Before sched_ext, a kernel scheduler regression meant either reverting (and losing new features and security patches) or accepting the degradation. Now there is a third path: write a workload-specific policy that runs alongside the upstream kernel. This decouples your optimization path from the kernel's evolution. You upgrade the kernel on your schedule, not theirs.

Experiment velocity compounds. The first sched_ext deployment took months of engineering. But the follow-up improvements shipped in days because the policy lives in user space. Each iteration revealed the next bottleneck. The cost of experimentation dropped from kernel-patch scope to binary-update scope. That transforms what is worth trying. When a change costs days instead of months, you try things you would never have attempted before.

Where This Applies

If you run latency-sensitive services on Linux, sched_ext is available in kernel 6.12 and later. Any operator with a workload that does not fit the general-purpose model can ship workload-specific scheduling policies without forking the kernel.

This is not just for Meta-scale operations. Web serving, API gateways, database engines, real-time media processing, and game servers all have scheduling patterns that differ from the general case. If your service has clear latency-critical vs background work, and that distinction is stable enough to encode, you are a candidate.

The threshold is lower than most teams assume. You do not need a dedicated kernel team. The BPF program for a simple scheduling policy can be written by a systems engineer who understands your workload. The complexity is in the policy logic, not in the kernel integration.

The Trade-offs

Writing a BPF scheduler still requires kernel expertise, careful testing, and ongoing maintenance. The payback depends on how much latency matters to your business. For most products, the default scheduler is fine. But the decision should be deliberate.

What Meta demonstrated is that the option now exists for everyone. Sched_ext was upstreamed into the Linux kernel and is available to any operator running kernel 6.12 or later. The question shifts from whether it is possible to whether your workload is special enough to justify it.