Observing Kubernetes Network Behavior with eBPF: Tracing Packet Lifecycle in Cilium Without Breaking Production

Traditional Kubernetes network debugging relies on tools like tcpdump, kubectl exec into pods, or adding temporary sidecar proxies — all of which introduce latency, consume resources, or fundamentally alter the traffic you're trying to observe. When a mysterious connection timeout hits a cluster running Cilium as its CNI, these methods barely scratch the surface. The packet has already traversed iptables rules, Cilium's BPF maps, endpoint policy enforcement hooks, and potentially service mesh layers before it reaches the application socket. By the time you capture it at the pod interface, the context of why it was allowed or denied has been lost. This is where eBPF changes everything — by giving you visibility directly inside the kernel's networking data path, at the exact point where decisions are made.

Cilium attaches hundreds of BPF programs to hook points across the kernel's networking stack. Every time a packet enters or leaves a pod's network namespace, traverses a service endpoint, or is evaluated against network policies, a BPF program intercepts it. Cilium exposes these hooks through its Observer component, which streams flow records to a collector like Hubble. What makes this remarkable is that none of these BPF programs modify packet behavior — they are purely observational. You get a complete trace of each packet's journey: source and destination IPs and ports, the identity of the sending and receiving pods, the network policy that was evaluated (or skipped), the datapath action taken (forwarded, dropped, redirected), and the latency at each decision point. For a production system where you cannot afford packet loss or increased jitter, this is as close to zero-impact observability as you can get.

To get started beyond Hubble's default dashboard, you can stream Cilium's flow logs directly into an OpenTelemetry Collector and export them to a time-series database like ClickHouse or VictoriaMetrics. The key is to filter at the BPF hook level so you're only capturing the connections you care about. Here's a practical configuration snippet for a Cilium Hubble relay pipeline:

# hubble-relay-config.yaml
relay:
  listenAddress: ':4245'
  tls:
    enabled: false
  output:
    - type: openTelemetry
      endpoints:
        - url: 'http://otel-collector:4317'
      format: json
      enableExtraMetadata: true
      enableSystemContext: true
      enableSourceKubernetes: true
      enableDestinationKubernetes: true

Once these flow records land in your observability backend, you can build queries that answer questions traditional tools simply cannot. For instance, you can correlate a sudden spike in TCP RST packets with a recent network policy update, or identify that a specific pod is being dropped at the ingress endpoint layer rather than at the application layer — saving hours of blame-shifting between platform and application teams. Combining this with Cilium's monitor buffer metrics helps you catch dropped flow records early, which is critical because even eBPF-based systems have buffer limits, and under extreme packet rates, you can miss events if the ring buffer backs up.

The real power emerges when you layer eBPF flow data on top of Cilium's identity model. Every pod in a Cilium-managed cluster gets a security identity — an opaque integer that represents its logical role in the network. When you see a flow record showing that identity 42 (say, the frontend service) is communicating with identity 87 (backend-api), and a drop event occurs, you immediately know it's a policy issue between those two logical roles — not a bug in IP addressing or DNS resolution. You can also trace L3/L4 handoffs, observe how Cilium's NodePort implementation bypasses iptables entirely via BPF hooks, and even measure the overhead of encrypting traffic with WireGuard by comparing latencies on encrypted vs. unencrypted paths. This level of kernel-deep insight transforms how you reason about network behavior: you stop guessing and start tracing, directly from the point where the Linux kernel makes its decisions. For teams running Cilium in production today, investing time in understanding the flow data model is one of the highest-ROI observability moves you can make.