Skip to main content
bex.co
419
Blog

Incus as a Cluster API Node Substrate: A Real Provider, a Narrow Case, and a Default You Should Flip

8 min readDora NodaDora Noda
Infrastructure
Self-Hosting
PaaS
Security
Share
On this page

There's a working Cluster API infrastructure provider for Incus — cluster-api-provider-incus, CAPN for short — and it will happily hand you kubectl get nodes output where every node is an LXC system container instead of a VM or a bare-metal box. It's real, it's maintained under the lxc GitHub org, and it supports kube-vip and OVN load balancers, IPAM, and both container and VM instance types out of the box.

It also defaults to launching every one of those nodes privileged. That single default — privileged: true unless you explicitly set PRIVILEGED=false in the cluster template — is the fact that should shape how a self-hosted PaaS thinks about this provider, more than any density benchmark or governance backstory. Here's what Incus system containers actually are, what CAPN buys a Cluster-API-managed Hetzner fleet, and where the case for exposing them to tenants is genuinely narrow rather than a general upgrade over the Kubernetes-Pod-plus-gVisor/Kata model bex already runs.

What Incus system containers actually are​

Incus is the community-governed fork of Canonical's LXD, created when Canonical pulled LXD maintainership in-house and the Linux Containers project forked the last community-controlled snapshot to keep it open. Two years on, Incus is packaged in Debian stable, Fedora, openSUSE, and NixOS; LXD remains Snap-only and Ubuntu-first. Incus ships under Apache 2.0 with no contributor license agreement, which is the detail that actually matters for a self-hosted platform picking dependencies — no single vendor can relicense or re-fork the ground out from under it the way Canonical did to the community once already.

A system container is not a Docker container with extra steps. It's a full Linux userspace — its own init system (typically systemd), its own persistent root filesystem, multiple long-running processes, SSH access if you want it — running under the host's kernel instead of a dedicated guest kernel. Incus also manages full VMs and single-process "application containers" through the same API, but the system container is the primitive that doesn't map cleanly onto either a Kubernetes Pod or a KVM guest: it's lighter than a VM because there's no hypervisor and no second kernel to boot, and it's heavier — in the sense of "does more" — than a Pod because it's a whole machine, not one process.

That lightness shows up directly in the numbers. Namespace-isolated containers on Linux run with roughly 1-2% overhead against bare metal; a KVM guest runs 5-20% overhead depending on workload, because every syscall a VM's guest kernel can't handle locally costs a VM-exit round trip to the host, and every VM carries a second kernel's memory footprint whether or not the workload needs it. On identical hardware, that difference compounds into density: benchmarks packing containers versus VMs on the same box report 5-8x more containers fitting in the same RAM budget for equivalent per-instance workloads. For a Hetzner-fleet operator whose entire cost model runs on bin-packing tenants onto owned boxes rather than renting hyperscaler instances, that's the number that makes Incus system containers worth a second look at all — a lightweight-VM-shaped primitive that doesn't carry VM-shaped overhead.

CAPN: the part that already exists, and the default worth flipping​

The natural objection to any "what if we ran a different node substrate" post is that it's speculative — nobody's actually wired it up. That objection doesn't apply here. cluster-api-provider-incus is a real Cluster API infrastructure provider, documented at capn.linuxcontainers.org, that provisions Kubernetes control-plane and worker nodes as Incus instances.

Set instanceType: container in the machine template and CAPN launches an LXC system container per node; set instanceType: vm and it launches a full VM instead — the same provider, the same API, your choice per machine pool. It handles the load-balancer problem a bare Incus cluster doesn't solve on its own, via kube-vip for production or OVN/haproxy for development, and it claims and releases load-balancer IPs itself through built-in IPAM. As of the most recent builds it tracks Cluster API v1.11's v1beta2 contract, and its images build against Ubuntu 24.04 and Debian 13.

The default worth flagging before anyone points this at a real fleet: CAPN launches container-backed nodes privileged by default. Incus itself defaults every container to unprivileged — running inside a user namespace, where root inside the container maps to an unprivileged UID on the host, so a container-to-host escape doesn't hand the attacker host root. CAPN overrides that default for its own cluster nodes because kubeadm and the container runtime need capabilities an unprivileged container doesn't have out of the box.

Unprivileged operation is supported — the project's own docs cover it, and pre-built kubeadm images have supported it since v1.32.4 — but it's opt-in via PRIVILEGED=false in the cluster template, not the path of least resistance. A team that clones the quick-start and never revisits that flag ships every node — control plane included — as a privileged container, which meaningfully narrows the isolation gap between "container escape" and "host root" that the rest of this analysis assumes you'd want closed.

Where the isolation boundary actually sits​

Here's the comparison that determines whether this is worth exposing to tenants at all: does an Incus system container, run correctly (unprivileged, via CAPN's opt-in flag), draw a stronger isolation boundary than the Kubernetes-Pod-plus-gVisor/Kata model a Cluster-API-managed Hetzner fleet already runs for untrusted tenant workloads?

No — and the mechanism explains why. An unprivileged Incus container still shares the host's kernel; the user-namespace mapping stops a privilege escalation (root-in-container becoming root-on-host) but does nothing about a kernel exploit, because the container's process is still making real syscalls into the one kernel every other container and the host itself are running on. That's the identical trust boundary a plain runc Kubernetes Pod draws.

Kata Containers exists specifically to close that gap by giving every workload its own guest kernel behind a hardware (KVM) boundary — a kernel bug in one tenant's workload can't reach a neighbor's, because they're not sharing a kernel to begin with. gVisor closes a narrower version of the same gap by interposing a userspace reimplementation of the Linux syscall surface between the workload and the real kernel, so most exploitable syscalls never reach it. An unprivileged Incus system container has neither property.

It's a real, useful hardening step over a privileged container — which is why CAPN's default matters — but it is not a peer of gVisor or Kata on the question of "can a hostile tenant reach the kernel other tenants depend on." Running Incus system containers instead of Kubernetes Pods for hostile-tenant isolation would be a downgrade from what a gVisor/Kata RuntimeClass already provides today, not an upgrade.

The narrow case where it's still worth something​

None of that makes Incus system containers pointless for a PaaS — it means the justification has to be a different axis than security. The genuine case is tenants who want a full machine, not a single process: an app that assumes it can ssh in, run a cron daemon and a web server and a background worker under one systemd, write to a persistent local filesystem outside a single container's lifecycle, or otherwise behave like it's sitting on a VPS rather than inside a Kubernetes Pod. That's a real, if narrow, slice of what a Render/Heroku-alternative's customer base includes — legacy apps lifted-and-shifted without a rewrite, or teams who explicitly want VPS-shaped ergonomics rather than a Kubernetes abstraction leaking through. For that tenant, the alternative isn't "Kubernetes Pod with gVisor" — it's "full KVM VM," and there an Incus system container is a legitimate lighter-weight offer: same full-OS ergonomics, a fraction of the VM's memory and boot-time overhead, and 5-8x the density on the same owned Hetzner hardware.

Verdict: not a second primitive to expose today​

Weigh the two sides concretely.

What it costs a fleet that already runs Cluster API on Hetzner:

  • A second node-image build pipeline alongside the existing Kubernetes-node images
  • A second CSI/storage compatibility surface to validate
  • A second network model to reconcile with the CNI the Kubernetes side already runs
  • A dependency on a provider that's still pre-1.0 (v0.9.x-era releases, v1alpha2/v1beta2 APIs still moving)

What it buys:

  • A better-than-VM answer for the narrow slice of tenants who need a full persistent Linux environment rather than a single-process container
  • Nothing at all for the isolation question that dominates most multi-tenant PaaS threat models — a gVisor/Kata RuntimeClass on the existing Kubernetes-Pod model already draws a stronger boundary than an Incus system container would

That's not a reason to ignore CAPN — it's a reason to watch it rather than adopt it. Worth prototyping behind a flag if "give me a full VM-like box, not a container" turns from a hypothetical into an actual support-ticket pattern; worth revisiting once CAPN clears 1.0 and the privileged-by-default template stops being the path of least resistance. Worth shipping as a general-purpose second isolation primitive today, on the evidence above, it is not.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, backed by a Cluster API-managed fleet. Star the repo on GitHub or deploy your first app today.

Sources​

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex
Back to all posts