eBPF: The Kernel Technology Reshaping Data Center Networking and Observability

What is eBPF?

eBPF (extended Berkeley Packet Filter) is a sandboxed virtual machine inside the Linux kernel that lets you run custom programs at kernel level — safely, without modifying kernel source or loading kernel modules. Think of it as JavaScript for the kernel: it decouples the pace of kernel innovation from the pace of infrastructure tooling, just like JS decoupled web application evolution from browser releases.

The kernel has always been the ideal place to implement networking, observability, and security because it sees everything — every packet, every syscall, every file access. But historically, extending the kernel meant either writing a kernel module (dangerous, version-specific) or convincing the LKML to merge your changes (slow, political). eBPF changes this by providing a safe, programmable layer that runs inline with kernel operations.

How It Works

eBPF programs are event-driven. You write them in C (compiled to eBPF bytecode via LLVM/Clang), load them via the bpf() syscall, and attach them to hook points in the kernel:

• Syscalls — trace every open(), read(), write(), connect()
• Network events — intercept packets at XDP (driver level), TC (traffic control), or socket level
• Tracepoints/kprobes — attach to any kernel function
• uprobes — attach to user-space functions

Before a program runs, it passes through the eBPF verifier — a static analyzer that guarantees the program will: (a) terminate (no loops without bounds), (b) not crash the kernel, (c) not access arbitrary kernel memory. Then the JIT compiler translates bytecode to native machine instructions, making it run as fast as natively compiled kernel code.

eBPF programs communicate with user space via eBPF maps — key-value stores, arrays, hash maps, ring buffers, and per-CPU variants. A program writes data to a map, and a user-space agent reads it. This is the plumbing behind every eBPF-based tool.

Why It Matters for Data Center Networking

This is where eBPF hits closest to what you do at iColo. Traditional Kubernetes networking uses iptables — a chain of rules evaluated sequentially. As your cluster grows, iptables rules explode (O(n) per packet), and performance degrades sharply. A cluster with 10,000 services can have over 50,000 iptables rules.

Cilium replaces iptables entirely with eBPF programs attached to the network stack. Instead of chaining rules, Cilium installs efficient eBPF programs that process packets in O(1) — a single hash lookup or direct decision at the XDP layer. The result is 30-40% higher throughput and significantly lower latency compared to iptables-based CNIs.

Cilium also handles:

• Load balancing — eBPF-based kube-proxy replacement that avoids iptables overhead
• Network policies — enforced at the socket layer, not just packet layer, meaning you get kernel-level verification of identity (based on service identity, not IPs)
• Bandwidth management — EDT (Earliest Departure Time) via eBPF for fair pod bandwidth
• Mutual TLS — Transparent proxy without sidecars

This is the same tech running at Meta, where eBPF processes and load-balances every packet entering their data centers. Cloudflare uses it for DDoS mitigation, packet filtering, and performance monitoring at the network edge.

Observability Without Instrumentation

Traditional observability requires adding OpenTelemetry agents, instrumenting code, injecting sidecars — all of which add complexity and resource overhead. eBPF achieves zero-instrumentation observability:

• Pixie (acquired by New Relic) uses eBPF to capture every HTTP request, database query, and function call across your cluster — without modifying a single line of application code. It sees everything because the kernel sees everything.
• bpftrace gives you a one-liner tracing language (awk-inspired) to probe any kernel or user-space function. Want to see every open() syscall by PID? bpftrace -e 'kprobe:do_sys_open { printf("%d %s\n", pid, str(arg1)); }'
• OpenTelemetry eBPF — there's ongoing work to use eBPF as an automatic telemetry source, generating traces and metrics without manual instrumentation.

For a self-hoster running Hermes on Arch, this means you can debug network issues, trace syscall latencies, and profile performance without rebuilding kernels or installing heavy tooling. bpftrace is probably already available in the Arch repos.

Security at Kernel Level

• Falco — the CNCF's runtime security tool — uses eBPF to monitor syscalls and detect anomalous behavior in real time. It can alert when a shell opens inside a container that shouldn't have one, when a sensitive file is read unexpectedly, or when a process spawns a child outside its expected tree.
• Tetragon — Isovalent's Cilium sibling — provides eBPF-based security observability and runtime enforcement. It can kill a process, block a network connection, or trigger an alert based on kernel-level signals, all from eBPF programs that execute inline with the event.

The beauty is all of this runs without sidecars, without agent containers in each pod's network namespace, and without modifying the application image.

The Trade-offs

eBPF isn't magic. The verifier is strict — no arbitrary loops, no unbounded memory access, limited stack depth (512 bytes). Writing non-trivial eBPF programs requires understanding kernel internals. BPF CO-RE (Compile Once, Run Everywhere) solves the cross-kernel-version problem but adds complexity. And for very advanced operations (hardware interaction, block device drivers), kernel modules are still the right tool.

But for the 80% case — packet processing, syscall tracing, observability, security enforcement — eBPF is the new baseline. It's the infrastructure layer beneath the infrastructure you already use.

Try It Today

On your Arch box:

# Check if eBPF is supported (kernel 5.4+)
uname -r  # should be 5.4+

# Install bpftrace
sudo pacman -S bpftrace

# Trace all new TCP connections
sudo bpftrace -e 'tcp:connect { printf("%s:%d -> %s:%d\n", ntop(2, arg0), arg1, ntop(2, arg2), arg3); }'

# See what exec() calls are happening system-wide
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s\n", str(args->filename)); }'

That last command shows you every binary being executed on your system in real time — without polling /proc, without auditd, without modifying anything. That's the power of eBPF.