Skip to main content

Cilium Tetragon Hits Production Maturity: What Kernel-Level eBPF Enforcement Adds to a Self-Hosted PaaS's Sandbox

8 min readDora NodaDora Noda
Share
On this page

A sandbox boundary answers one question: can this container's syscalls reach the host kernel. It says nothing about what a permitted process does once it's running inside that boundary — a tenant's legitimate-looking build step that quietly forks a cryptominer, or an escape attempt in the half-second before the sandbox actually stops it. That gap is what Cilium Tetragon is built to close, and as of its 1.4 release in February 2026, it closed it well enough that CNCF's 2026 Observability TAG survey found 67% of teams running Kubernetes at scale already had at least one eBPF-based observability tool in production. For a self-hosted PaaS running untrusted tenant workloads on owned Hetzner nodes, the question isn't whether to add kernel-level visibility on top of sandbox isolation — it's whether Tetragon's enforcement mode, not just its observability side, is mature enough to trust with a SIGKILL on a real tenant process.

What Tetragon actually watches

Tetragon is a Kubernetes-aware security tool built on eBPF that hooks directly into kernel syscalls — execve, file opens, network connects — and correlates every event back to the pod and container that produced it, without a sidecar per pod and without routing traffic through a userspace proxy. Three event types cover most of what a multi-tenant platform needs:

  • Process execution, with full process lineage (parent chain, not just the leaf binary) and the exact command line
  • File access below the audit subsystem — the kernel sees a file open before auditd would ever log it
  • Network connections, attributed to the specific process that opened the socket, not just the pod IP

The distinguishing property is where the filtering happens. Tetragon's selectors run inside the eBPF program itself, in-kernel, before an event is even considered for export to userspace. An event that doesn't match any policy never crosses the kernel/userspace boundary at all. That's the architectural difference from a tool like Falco, which captures syscalls via eBPF or a kernel module but evaluates its rules in a userspace agent — every event crosses over first, rules run after. In one production comparison running roughly 40 active TracingPolicies, Tetragon measured about 0.7% CPU overhead at steady state against Falco's 1.4% for comparable coverage — not because Tetragon is a faster rules engine, but because it's throwing away the events it doesn't care about before they leave the kernel.

Defense in depth: the sandbox boundary and the kernel watchdog aren't the same layer

This site already covers the sandbox-isolation decision — gVisor's user-space kernel versus Kata's microVM boundary as the RuntimeClass choice for isolating untrusted tenant workloads from the host. That coverage treats sandboxing as the first line of defense, and it's the right first line: it shrinks the syscall surface a malicious container can even attempt to reach. But a sandbox boundary is a perimeter, not a behavior monitor. Two failure modes fall outside what gVisor or Kata alone catch:

  1. A process that's fully permitted inside the sandbox but shouldn't be running at all. A tenant's build container spawning xmrig or a stratum-protocol miner binary isn't a syscall the sandbox needs to intercept — every syscall it makes is one the container is allowed to make. The sandbox has nothing to say about it. Something has to watch process identity, not just syscall surface.
  2. The moment of an escape attempt itself. If a container does find a path past the sandbox — a mount or nsenter call chained with a capability it shouldn't have — the value is in catching that specific sequence as it happens, not reconstructing it from logs after the node is already compromised.

Tetragon sits on the layer sandboxing doesn't cover: not "can this syscall reach the kernel" but "is this specific process, with this lineage, doing something it's never supposed to do." Running both isn't redundant — it's the difference between a locked door and a camera pointed at what happens on both sides of it.

The network-attribution piece matters for the same reason. A sandbox boundary doesn't distinguish a tenant's application making an outbound API call from that same pod's process quietly opening a connection to a mining pool or a C2 endpoint — both are "a syscall the container is allowed to make." Tetragon's per-process network events answer a different question than a CNI-level flow log does: not just which pod talked to which destination, but which specific binary, with which parent process, opened the socket. That's the detail that turns "pod X made an outbound connection at 03:14" into "the xmrig binary that sys_execve already should have killed made an outbound connection at 03:14" — a correlation a flow log alone can't produce, because it never knew which process owned the packet.

A concrete enforcement policy

Here's what that looks like as an actual TracingPolicy, not a slide. This one does two things a multi-tenant PaaS running on shared Hetzner nodes cares about immediately: it kills any process inside a tenant container that calls mount while holding CAP_SYS_ADMIN (the capability + syscall pairing behind most documented container-escape techniques, including nsenter-based host access), and it kills any process whose exec path matches a known cryptominer binary pattern.

yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "tenant-escape-and-miner-enforcement"
spec:
  kprobes:
    - call: "sys_mount"
      syscall: true
      selectors:
        - matchCapabilities:
            - type: Effective
              operator: In
              values:
                - "CAP_SYS_ADMIN"
          matchActions:
            - action: Sigkill
    - call: "sys_execve"
      syscall: true
      selectors:
        - matchBinaries:
            - operator: In
              values:
                - "/usr/bin/xmrig"
                - "/tmp/xmrig"
                - "/usr/bin/minerd"
          matchActions:
            - action: Sigkill

Both selectors run as Sigkill actions evaluated in-kernel — the process is terminated as part of the syscall path, not flagged for a controller to reconcile away seconds later. That's the practical meaning of "kernel-level enforcement": the difference between a tenant's cryptominer running for the 30 seconds it takes a log pipeline to notice, and it never completing its first execve.

The scale this holds up at is documented, not theoretical: one production Tetragon deployment across a 4,200-node fleet reported roughly 180GB of Tetragon events per day after deduplication, running around 40 active TracingPolicies at that 0.7% steady-state CPU cost cited above. For an operator running owned Hetzner hardware rather than renting managed node pools, that overhead is a rounding error against the CPU already paid for — there's no per-agent licensing meter and no sidecar-per-pod tax to multiply across a fleet.

Is enforcement mode actually production-ready?

This is the question the title promises an honest answer to, and the honest answer is conditionally yes — the maturity gain in the 1.4 release (February 2026) was specifically in policy authoring: selector syntax, matchBinaries/matchCapabilities ergonomics, and the rough edges that made writing a correct TracingPolicy error-prone in earlier releases. It was not a claim that enforcement is safe to flip on for a policy the moment you write it.

The recommended rollout is staged, and skipping the stages is how enforcement mode turns into a self-inflicted outage:

  1. Write the policy with Post actions only (observe, don't kill) and run it for at least two weeks per cluster.
  2. Export the events to your SIEM or log pipeline and build a dashboard of what actually matches.
  3. Tune out false positives — a legitimate CI runner that happens to call mount for a build cache is a false positive you'll only find by watching it fire.
  4. Only then flip the matching selector's action from Post to Sigkill.

That's not a caveat that undercuts kernel-level enforcement — it's the same operational discipline any admission-control or auto-remediation system needs before it's allowed to kill something in production. What changed with 1.4 is that authoring the policy correctly the first time got materially easier, which shortens the observe-only phase instead of eliminating it. The same release window, Cilium 1.19 shipped as the project's ten-year release with a hardening focus of its own — strict IPsec/WireGuard modes that now drop unencrypted inter-node traffic outright instead of merely logging it — which matters here because Tetragon ships as part of the same Cilium project and inherits that hardening posture on the network layer underneath the process-level policies above.

What this buys a self-hosted PaaS

For a platform running untrusted tenant containers on owned nodes rather than someone else's managed control plane, the sandbox boundary and the kernel watchdog answer different questions, and a managed PaaS renting fixed-size instances typically gives an operator neither lever — you get whatever isolation and observability the vendor decided to expose, not a TracingPolicy you can write yourself. Owning the node is what makes "author the enforcement policy for my threat model" an option instead of a support ticket.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex