Deep Kernel Observability: Building an eBPF-Based Heap Integrity Monitor
Deep Kernel Observability: Building an eBPF-Based Heap Integrity Monitor
Heap corruption bugs—buffer overflows, use-after-free, and double-free errors—are among the most insidious failures in production systems. They don't just crash your service; they corrupt adjacent allocations, leak sensitive data, and create exploitable attack surfaces. Yet, most production monitoring stacks ignore kernel-level memory behavior entirely.
While tools like Valgrind or AddressSanitizer are indispensable during development, they are non-starters in production. The overhead is prohibitive, and you cannot attach a debugger to a live gRPC server serving 50,000 requests per second without catastrophic impact.
What if you could observe kernel memory allocation patterns—kmalloc, kfree, and kmem_cache_alloc—in real time, with near-zero overhead, using eBPF?
In this guide, I will walk you through building a production-safe heap integrity monitor that detects double-free events, flags suspicious allocation patterns, and exports structured findings to userspace—all without modifying a single line of application code or restarting a single pod.
The Architecture
To monitor memory without impacting the stability of the host, we must use a non-intrusive approach. We attach probes to kernel memory management functions, maintain an in-kernel map of every live allocation, and validate each deallocation against that map.
┌─────────────────────────────────────────────────────┐
│ Host Kernel (eBPF) │
│ │
│ ┌──────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Kprobe on │ │ Kretprobe on │ │ Ring Buffer │ │
│ │ kmalloc │ │ kfree │ │ (perf_event) │ │
│ │ __kmalloc │ │ kmem_cache │ │ │ │
│ │ kmem_cache│ │ _free │ │ │ │
│ └─────┬─────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────────┐ │
│ │ Allocation Tracker Map (BPF_MAP_TYPE_ │ │
│ │ HASH) │ │
│ │ Key: ptr (u64) │ │
│ │ Value: AllocationMeta { │ │
│ │ size, pid, comm[16], tstamp, │ │
│ │ call_site } │ │
│ └──────────────────────────┬──────────────────┘ │
│ │ │
└─────────────────────────────┼────────────────────────┘
│ perf_buffer_mmap()
▼
┌─────────────────────────────────────────────────────┐
│ Userspace Agent (Rust) │
│ │
│ ┌──────────────┐ ┌─────────────────────────────┐ │
│ │ Perf Reader │ │ Anomaly Detector │ │
│ │ (libbpf) │ │ - double-free detection │ │
│ │ │ │ - use-after-free heuristics │ │
│ └──────┬───────┘ └────────────┬────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────────────────────────────────┐ │
│ │ Alert Exporter → OpenTelemetry / Prometheus │ │
│ │ / Structured Log (JSON to syslog) │ │
│ └─────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
Step 1: The eBPF Program — Probing kmalloc and kfree
We use kprobes to intercept function arguments and kretprobes to capture return values. A critical challenge in eBPF is that __kmalloc returns the allocated pointer only upon completion. Therefore, we must use a two-step process: capture the requested size in a kprobe, and register the returned pointer in a kretprobe.
To handle recursive allocations (where a function calls kmalloc inside another kmalloc call), we use a BPF_MAP_TYPE_PERCPU_HASH keyed by pid_tgid. This prevents race conditions and overwrites on the same CPU.
// heap_monitor.bpf.c
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
struct alloc_args {
u64 size;
u64 flags;
};
struct alloc_meta {
u64 size;
u64 pid_tgid;
u64 tstamp_ns;
char comm[16];
u64 call_site;
};
// Per-CPU temp storage to handle recursive calls safely
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_HASH);
__uint(max_entries, 1024);
__type(key, u64); // pid_tgid
__type(value, struct alloc_args);
} pending_allocs SEC(".maps");
// The primary tracker for live allocations
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 1 << 20);
__type(key, u64); // pointer address
__type(value, struct alloc_meta);
} alloc_tracker SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
__uint(max_entries, 64);
} events SEC(".maps");
struct anomaly_event {
u64 ptr;
u64 pid_tgid;
u64 tstamp_ns;
char comm[16];
char anomaly_type[32];
char details[128];
};
// 1. Capture allocation arguments
SEC("kprobe/__kmalloc")
int BPF_KPROBE(__kmalloc_probe, size_t size, gfp_t flags)
{
u64 pid_tgid = bpf_get_current_pid_tgid();
struct alloc_args args = {.size = size,.flags = (u64)flags };
bpf_map_update_elem(&pending_allocs, &pid_tgid, &args, BPF_ANY);
return 0;
}
// 2. Capture the returned pointer and register it
SEC("kretprobe/__kmalloc")
int BPF_KRETPROBE(__kmalloc_ret, long ret)
{
if (ret <= 0) return 0;
u64 pid_tgid = bpf_get_current_pid_tgid();
struct alloc_args *args = bpf_map_lookup_elem(&pending_allocs, &pid_tgid);
if (!args) return 0;
struct alloc_meta meta = {
size = args->size,
pid_tgid = pid_tgid,
tstamp_ns = bpf_ktime_get_ns(),
call_site = (u64)BPF_KRETPROBE_RETURN_ADDRESS,
};
bpf_get_current_comm(&meta.comm, sizeof(meta.comm));
u64 ptr_val = (u64)ret;
bpf_map_update_elem(&alloc_tracker, &ptr_val, &meta, BPF_ANY);
bpf_map_delete_elem(&pending_allocs, &pid_tgid);
return 0;
}
// 3. Validate deallocations
SEC("kprobe/kfree")
int kfree_probe(struct pt_regs *ctx)
{
void *ptr = (void *)PT_REGS_PARM1(ctx);
if (!ptr) return 0;
u64 ptr_val = (u64)ptr;
u64 pid_tgid = bpf_get_current_pid_tgid();
struct alloc_meta *meta = bpf_map_lookup_elem(&alloc_tracker, &ptr_val);
if (!meta) {
// If the pointer isn't in our tracker, it's a double-free or UAF
struct anomaly_event evt = {
ptr = ptr_val,
pid_tgid = pid_tgid,
tstamp_ns = bpf_ktime_get_ns(),
};
bpf_get_current_comm(evt.comm, sizeof(evt.comm));
__builtin_memcpy(evt.anomaly_type, "invalid_free", 13);
__builtin_memcpy(evt.details, "Pointer not in tracker (Double-free/UAF)", 40);
bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU, &evt, sizeof(evt));
} else {
bpf_map_delete_elem(&alloc_tracker, &ptr_val);
}
return 0;
}
char _license[] SEC("license") = "GPL";
Step 2: The Userspace Agent — Rust with Aya
To process these events, we use the aya crate. It provides a high-performance, asynchronous interface to BPF maps and perf buffers. Because we are monitoring kernel-level events, the userspace agent must be efficient to avoid dropping events from the ring buffer.
// src/main.rs
use aya::maps::PerfEventArray;
use aya::programs::{KProbe, KRetProbe};
use aya::Bpf;
use std::mem::MaybeUninit;
mod ebpf {
aya::include_ebpf!("heap_monitor.o");
}
#[repr(C)]
struct AnomalyEvent {
ptr: u64,
pid_tgid: u64,
tstamp_ns: u64,
comm: [u8; 16],
anomaly_type: [u8; 32],
details: [u8; 128],
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut bpf = Bpf::load(ebpf::HEAP_MONITOR.to_bytes())?;
// Attach probes
let kprobe_kmalloc: &mut KProbe = bpf.program_mut("__kmalloc_probe")?.try_into()?;
let kretprobe_kmalloc: &mut KRetProbe = bpf.program_mut("__kmalloc_ret")?.try_into()?;
let kprobe_kfree: &mut KProbe = bpf.program_mut("kfree_probe")?.try_into()?;
kprobe_kmalloc.load()?.attach_kprobe("__kmalloc")?;
kretprobe_kmalloc.load()?.attach_kretprobe("__kmalloc")?;
kprobe_kfree.load()?.attach_kprobe("kfree")?;
let mut perf_array = PerfEventArray::try_from(bpf.map_mut("events")?)?;
let mut buffer = [MaybeUninit::<u8>::uninit(); 4096];
loop {
let events = perf_array.read_events(&mut buffer)?;
for event in events {
let evt = unsafe { std::ptr::read(event.as_ptr() as *const AnomalyEvent) };
let pid = evt.pid_tgid >> 32;
let comm = std::str::from_utf8(&evt.comm).unwrap_or("unknown").trim_matches('\0');
let detail = std::str::from_utf8(&evt.details).unwrap_or("").trim_matches('\0');
println!(
"[ANOMALY] Type: {}, PID: {}, Task: {}, Ptr: 0x{:x}, Detail: {}",
std::str::from_utf8(&evt.anomaly_type).unwrap_or("").trim_matches('\0'),
pid, comm, evt.ptr, detail
);
}
}
}
Step 3: Production Deployment
Running this in a Kubernetes cluster requires a DaemonSet. Since we are attaching to kernel symbols, the container requires elevated privileges. For security, we limit the capabilities to exactly what is needed and use a readOnlyRootFilesystem.
# heap-monitor-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: heap-monitor
namespace: observability
spec:
selector:
matchLabels:
app: heap-monitor
template:
metadata:
labels:
app: heap-monitor
spec:
hostPID: true
hostNetwork: true
containers:
- name: heap-monitor
image: internal-registry/heap-monitor:v1.0.0
securityContext:
privileged: true
capabilities:
add: ['BPF', 'SYS_ADMIN']
readOnlyRootFilesystem: true
resources:
limits:
cpu: '500m'
memory: '512Mi'
requests:
cpu: '100m'
memory: '256Mi'
Performance and Safety Considerations
1. Map Pressure and Eviction:
The alloc_tracker map is a BPF_MAP_TYPE_HASH. In high-load environments, the map may fill up. We use BPF_ANY for updates, meaning if the map is full, the update might fail. In production, it is vital to monitor the bpf_map_current_usage via Prometheus to ensure your map size (max_entries) is sufficient for your node's workload.
2. The "Pre-Free" Hook:
Note that kprobe/kfree intercepts the function before the kernel executes the deallocation logic. This is intentional. We want to validate the pointer while the memory is still valid to avoid race conditions where the memory is reclaimed before we can inspect it.
3. Distinguishing Double-Free from UAF:
The current implementation flags any kfree on an untracked pointer as an anomaly. To distinguish between a double-free and a use-after-free (UAF), you can implement a "tombstone" map. When a pointer is successfully freed, add it to a freed_ptrs map with a short TTL. If a kfree hits a pointer already in the freed_ptrs map, it is a double-free. If it hits an untracked pointer that was never seen being allocated, it is likely a use-after-free.
Summary
By moving memory integrity monitoring into the kernel via eBPF, we gain visibility that was previously impossible without significant performance penalties. This approach provides a foundation for advanced security features, such as slab cache analysis and automated root-cause analysis for memory corruption, turning the kernel from a "black box" into a transparent, auditable component of your observability stack.