eBPF Ate the Sidecar: What Hubble, Pixie, and Coroot Actually Replace on a Self-Hosted Fleet

Two programs observe every packet your tenants send. One of them runs once per pod, eats 250MB of RAM per copy, and adds milliseconds of latency to every request path. The other runs once per node, lives in the kernel, and costs roughly 5 millicores per pod. The CNCF Observability TAG reports that 67% of teams running Kubernetes at scale already run at least one eBPF-based observability tool in production — and the sidecar-per-pod telemetry model is what's being replaced.

So here is the verdict up front, for anyone operating a self-hosted multi-tenant fleet: run Coroot Community Edition as your default full-stack observer, keep Hubble on if Cilium is already your CNI, and reach for Pixie when you need to debug inside the cluster without shipping data anywhere. That is the whole post in one paragraph — the rest is the overhead math, what each tool actually replaces, and the boundary lines where eBPF still doesn't reach:

What the sidecar model actually costs​

The classic way to get per-request telemetry was a proxy next to every workload: an Envoy sidecar for metrics and traces, a logging agent sidecar for stdout shipping, sometimes an APM agent injected into the runtime. Each one is small. Multiplied by a fleet, they are a node tier.

Production breakdowns put the Envoy sidecar at roughly 250MB of resident memory per pod once mTLS and L7 filters are on, with measurable per-hop latency: about 3.2ms at p95 across a two-proxy request path, and up to 166% added mTLS handshake latency in the worst-measured configurations. A widely cited 2025 cluster accounting exercise tallied the sidecar fleet alone at around 80 GiB of RAM and 50 vCPUs of baseline CPU on a mid-size cluster — proxies doing nothing for the application itself. Scale it linearly and the shape is brutal:

That is the "up to 90% monitoring overhead reduction" claim made concrete: the telemetry cost stops scaling with tenant count and starts scaling with node count. On a multi-tenant PaaS where you pack dozens of small tenant services per node, per-pod overhead is a density tax your tenants pay in both bin-packing efficiency and p95 latency. A per-node kernel agent flips the curve — Cilium's WireGuard encryption path, for comparison, costs on the order of 5 millicores per pod, a 10–15x reduction from Envoy sidecar overhead for the L4 security piece.

The ecosystem already voted on the networking half of this: Istio's ambient mesh went stable (1.22–1.24) by moving L4 into a per-node ztunnel daemon, cutting mesh memory by up to 90% versus sidecars. eBPF observability is the same architectural move applied to telemetry — observe from the kernel once per node instead of proxying once per pod.

What each tool actually replaces​

The three names get lumped together as "eBPF observability," but they replace different things and demand different backends. Picking wrong means running a second stack six months later.

Hubble: your CNI's flight recorder​

Hubble is the observability layer of Cilium, and that sentence is both the pitch and the prerequisite. If Cilium is already your CNI, Hubble sees every TCP connection, HTTP request, DNS query, and dropped packet on every node with no application changes — enable the relay and UI with two Helm values and you have cluster-wide service maps and flow logs. What it replaces: the network-visibility half of a sidecar mesh and most of what teams bolted NetFlow-style collectors on for.

What it does not replace: application-level tracing, logs, or profiling. Hubble stops at roughly L7 metadata — it tells you the request happened and how long it took, not what the handler did with it. And the entry price is real: Hubble without Cilium as the CNI is not a meaningful deployment. If you run Calico, Flannel, or a cloud CNI, adopting Hubble means a CNI migration first, which is a much bigger decision than adopting an observer.

Pixie: a debugger that lives in the cluster​

Pixie (Pixie Labs, acquired by New Relic in 2020, open-sourced under Apache-2.0 and contributed to the CNCF as a sandbox project) takes the opposite approach from Hubble: full-body request capture, application profiles, and resource metrics, queried in-cluster with its PxL script language. The defining property is data gravity — telemetry is stored and queried inside your cluster, and nothing moves off-cluster unless you send it. What it replaces: the "add instrumentation, redeploy, reproduce, stare at dashboards" loop for live debugging, plus the APM agent for auto-instrumented languages.

The maturity caveat is worth naming plainly: Pixie is CNCF sandbox, not graduated, and its center of gravity has always been debugging workflows rather than long-retention monitoring. It is the tool you open during an incident, less obviously the system of record you alert from. For a platform team that has to run and patch the stack itself, that means Pixie complements a metrics backend rather than replacing one.

Coroot: the whole stack in one Helm chart​

Coroot is the closest thing to "replace the entire sidecar telemetry model with one thing." A per-node eBPF agent collects metrics, container logs, traces, and continuous profiles; the server stores them in Prometheus and ClickHouse and renders service maps, SLO tracking, and incident inspection in a single UI.

The Community Edition is self-hostable, and real-world self-hosted rollouts pair it with an embedded Prometheus with short retention (a couple of days, tens of GiB) precisely so Coroot's high-cardinality eBPF metrics never touch the main monitoring Prometheus — plus a single-replica ClickHouse with a few days of TTL for logs, traces, and profiles.

What it replaces: the Prometheus exporter set, the log shipper, the tracing SDK, and the profiler agent — four agents per pod collapsed into one agent per node. The price is running state: Coroot is a stateful application, and you are now the operator of its ClickHouse. For a team already running Postgres or Prometheus with persistent volumes, that is a familiar kind of work. For a team that wanted observability with no state to babysit, it is the honest tradeoff to name: kernel-level collection is nearly free, but somebody still stores the data.

Head to head: the pick for a team that patches its own stack​

The decision tree is short. If Cilium is your CNI, turn Hubble on — the marginal cost is near zero and flow visibility this cheap has no substitute. If your incidents start with "show me the actual request body on that pod," add Pixie next to whatever metrics backend you run.

And if you are choosing the one stack a small platform team will operate for the next two years, choose Coroot: its backends are Prometheus and ClickHouse, which means your existing runbooks, retention math, and volume backups transfer directly, and its cardinality is fenced by giving it its own short-retention Prometheus rather than sharing yours.

Note what is deliberately absent from this table: Grafana Beyla and the OpenTelemetry eBPF Instrumentation (OBI) beta shown at KubeCon EU 2026. Both point at the same zero-code future with OTel-native output, and OBI is arguably where the ecosystem converges — but a platform team choosing what to run this quarter should pick from what is operable today, and revisit OTel eBPF at GA.

What eBPF still doesn't give you​

Four boundary lines, checked before you commit:

• Kernel floor. Coroot's agent wants Linux 5.1+; full CO-RE portability across node image variants realistically wants 6.1+. If your fleet still runs older or mixed kernels — common on long-lived bare-metal nodes — audit uname -r across the pool before promising zero-instrumentation coverage.
• It observes; it does not proxy. Retries, timeouts, circuit breaking, and traffic shaping still need a mesh or gateway layer. eBPF tells you the retry storm is happening; Envoy (or ambient ztunnel plus waypoints, or Gateway API policy) is what stops it.
• Cardinality is still your bill. Per-request capture at fleet scale is a storage problem wearing a collection costume. Size ClickHouse TTLs in days, not months; isolate high-cardinality eBPF metrics in their own Prometheus; alert on the observer's own disk before your tenants do it for you.
• Encrypted payloads stay opaque. eBPF sees syscalls and socket buffers, not plaintext inside mTLS sessions it doesn't terminate. Service-mesh-encrypted traffic still needs proxy-level or app-level spans for body inspection — one more reason Pixie-style capture and mesh telemetry complement rather than duplicate each other.

The sidecar era is over; the storage era is the work now​

The 67% number marks the moment zero-instrumentation collection stopped being the exotic choice. The sidecar tax — hundreds of megabytes per pod, milliseconds per request path, a node tier of proxies — was worth paying when the alternative was flying blind. It is not worth paying now that the kernel will tell you the same things once per node.

For a self-hosted PaaS, that reframes the observability decision entirely. The question is no longer "which agents do we inject into tenant pods" — tenant pods should carry zero telemetry weight, which is also better tenancy hygiene. The question is "which stateful backend are we willing to operate," and Prometheus plus ClickHouse is the most boring, best-understood answer available. Start there, add Hubble if Cilium is your CNI, add Pixie for incident debugging, and keep an eye on OTel eBPF for the convergence play.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• CNCF Observability TAG survey via OpenTelemetry eBPF Instrumentation (OBI) guide, KubeCon EU 2026 — 67% of production clusters run at least one eBPF observability tool
• eBPF in Production Kubernetes: Ditch Your Sidecars in 2026 — 75GB sidecar RAM to 12GB case study
• The Real Cost of a Service Mesh: Istio Sidecar Overhead in Production — 250MB+/pod, 80 GiB + 50 vCPU cluster accounting, ~3.2ms p95, ~5m CPU/pod Cilium WireGuard vs Envoy
• Sidecars Are Dying in 2025: Ambient Mesh, eBPF, and Gateway API — ambient ztunnel 50–150MB/node, up to 90% memory reduction
• Coroot customer story: Arie's Adventures with Coroot — eBPF node-agent into Prometheus + ClickHouse, kernel 5.1+ floor
• New Relic open-sources Pixie; CNCF sandbox contribution — Apache-2.0, in-cluster observability
• 8 Best Kubernetes Pod Monitoring Tools in 2026 (Metoro) — Pixie/Cilium-Hubble/Coroot positioning cross-check