Hard Multi-Tenancy for Agent Workloads: What Apple's Kata plus VirtualClusters Recipe Costs

AI agents write and run code you did not review, in processes you did not start, with network access you did not personally approve. Put that workload on the same Kubernetes nodes as your paying tenants' production apps, separated by nothing but namespaces, and one prompt-injection or rogue curl | bash away sits a shared kernel — the one thing every container on the node has in common. The fix Apple demonstrated at KubeCon NA 2022 still sets the terms of the debate: give each tenant its own virtual control plane provisioned declaratively with Cluster API, and run the untrusted workloads themselves in Kata microVMs so a compromised sandbox gets its own kernel instead of sharing yours.

The price is concrete and measurable: roughly a second of cold-start latency and 30–100 MB of RAM per sandbox, plus one small virtual control plane per tenant. This post prices both layers, shows what each one contains, and gives the rule for when to pay.

Namespaces are a filing system, not a security boundary​

Soft multi-tenancy — namespaces plus RBAC, quotas, NetworkPolicies, and Pod Security Admission — is how trusted teams share a cluster. It is cheap, well-understood, and genuinely sufficient when every tenant is your own engineers. It fails as a security boundary in two places that matter for agent workloads.

First, the kernel is shared. Every container on a node makes syscalls into the same kernel, so a kernel exploit in one tenant's pod is a kernel exploit against the node and every neighbor on it. Namespaces scope visibility; they do not scope the kernel. Second, the control plane is shared. One API server serves every tenant, so a tenant that can reach the API server at all is one authentication bug away from somebody else's objects. The industry vocabulary for this is settled: namespaces give you soft multi-tenancy for tenants you trust, and untrusted tenants need hard multi-tenancy — separate control planes, separate kernels, or both.

Agent-generated code is the textbook untrusted tenant. It arrives as output from a model, it executes shell commands and installs packages as a matter of routine, and prompt injection means even a careful operator cannot vouch for what it will try next. Treating it like one more microservice in the agents namespace is filing it correctly while securing nothing.

Layer 1: a control plane per tenant, from Apple's recipe​

At KubeCon North America 2022, Apple's Chris Hein and Eric Ernst presented Running Isolated VirtualClusters With Kata & Cluster API: hard multi-tenancy achieved by running VirtualCluster control planes inside a host Kubernetes cluster, provisioning them declaratively with Cluster API Nested, and isolating the actual workloads with the Kata runtime plus virtual networking.

Each layer in that stack isolates a different thing, and keeping them straight is the whole design:

• Cluster API owns machine lifecycle declaratively — tenant clusters are Kubernetes objects reconciled by controllers, not tickets to an infra team. It answers "who provisions and upgrades the tenant's cluster."
• The virtual control plane (VirtualCluster in Apple's talk; vCluster is the widely-deployed equivalent, with Loft Labs reporting 40 million-plus virtual clusters created) answers "who can see and touch whose objects." From the host cluster's perspective a virtual cluster is just an application running in a namespace — its own API server, its own etcd-backed state — so a tenant with full admin inside its virtual cluster still has zero reach into the host API server or any sibling tenant.
• Kata plus virtual networking answers "who shares a kernel." Even inside one tenant's virtual cluster, the untrusted pods each get their own lightweight-VM kernel rather than sharing the node's.

The CNCF's own framing of the options puts this on a spectrum: namespaces and quotas at the soft end, virtual control planes in the middle, and a full separate physical cluster per tenant at the hard extreme. Apple's recipe lands one notch below separate physical clusters on isolation while staying far below them on cost — which is exactly the trade a shared fleet wants.

Layer 2: what a Kata sandbox actually contains​

Kata Containers look and feel like ordinary containers — same OCI image, same kubectl workflow, one runtimeClassName: kata on the pod — but underneath, each pod boots in its own lightweight virtual machine with its own kernel. The original motivation, per the project's own history, was precisely that namespace-based isolation "doesn't provide a strong barrier," which breaks multi-tenancy the moment tenants stop trusting each other.

Make it concrete with the scenario this whole design exists for. An agent sandbox executes a compromised build script that escalates to root and then attempts a container escape via a kernel vulnerability:

• On runc: root in the container plus a kernel bug equals code execution on the node, with every neighboring tenant pod — including production apps — in the blast radius. The shared kernel is a single point of failure for the entire node.
• On Kata: root in the container plus the same kernel bug equals code execution inside a throwaway microVM kernel that runs nothing else. The attacker owns a VM whose only tenant was already the attacker. The node kernel, the host control plane, and every neighbor pod are untouched. Deleting the pod deletes the entire compromised universe.

That containment is why Kata 4.0, released in 2026, was explicitly positioned as agent infrastructure: the Go runtime was rewritten in Rust (runtime-rs is now the default), improving memory safety and shrinking footprint and startup latency, and Kata is now one of the supported runtimes for Agent Sandbox, the Kubernetes SIG Apps project building a declarative Sandbox CRD for exactly these stateful, singleton agent workloads. Ant Group already runs Kata first for batch workloads and then for core online services, using it for real-time auditing of malicious activity. The ecosystem has converged: untrusted agent code gets its own kernel.

The price tag, measured: Kata's per-sandbox cost​

Isolation is never free, and Kata's costs land in two places — startup latency and memory — while a third feared cost, steady-state throughput, mostly evaporates on measurement.

Cold start. Booting a microVM takes longer than forking a namespaced process. Community benchmarks across hypervisors tell a consistent story: runc starts in roughly tens to a few hundred milliseconds, Kata on QEMU adds on the order of a second cold (around 200 ms warm), and Kata on Firecracker or Cloud Hypervisor splits the difference at roughly 600 ms cold and 125 ms warm. Agent-sandbox-shaped benchmarks report p50 sandbox startup near 400 ms, p95 near 900 ms, and p99 near 1.5 s against runc's ~80/200/350 ms at the same percentiles. The honest summary: budget about a second extra on cold start, less on warm, least on Firecracker-class hypervisors.

Memory. Each microVM carries its own kernel and guest footprint: roughly 30–100 MB per sandbox depending on hypervisor and configuration (about 30 MB with lean Cloud Hypervisor setups, up to 50–100 MB per VM with QEMU-flavored ones, versus ~10–20 MB for something like gVisor). For a fleet packing a hundred concurrent agent sandboxes, that is single-digit gigabytes of overhead — visible on the capacity plan, not dominant on it.

Steady state. Once running, the virtualization tax is small: compute-bound work runs at near-native speed, workspace file I/O lands around 95% of native over virtio, and network latency adds on the order of 50–150 microseconds per packet. One head-to-head benchmark suite summarized it well: cold start ranges from ~20 ms to ~500 ms across runtimes, steady-state request latency barely differs, and the real cost is memory and compatibility, not throughput. Do not let anyone veto Kata over steady-state performance; the veto-worthy numbers are all at startup.

Layer 1's price tag: a virtual control plane costs a fixed amount per tenant​

Here is the cost shape the Kata table hides: Kata's overhead scales per pod, but a virtual control plane's overhead scales per tenant. Each virtual cluster is a small deployment on the host — its own API server, a syncer reconciling objects down into the host namespace, an embedded store — with an idle footprint measured in fractional CPU and hundreds of megabytes of RAM. That is trivially cheap next to the alternative it replaces (dedicated control-plane nodes and a full etcd quorum per tenant), but it is not zero, and it has a second component that does not show up in resource requests: every virtual control plane is one more API server to upgrade, monitor, and back up.

The practical consequence is a crossover. A handful of tenants with many sandboxes each is the sweet spot: the per-tenant control-plane cost amortizes to nearly nothing while Kata's per-sandbox cost is exactly where it protects you. Hundreds of tenants with one pod each inverts the math — at that point the fleet of virtual API servers costs more, operationally, than the isolation it buys, and the honest answer is fewer, coarser tenants or separate physical clusters for the genuinely adversarial ones. Price the two layers separately or you will misprice the system: Kata bills per sandbox, virtual clusters bill per tenant, and the tenant count — not the pod count — is what sizes layer 1.

There is also a hard prerequisite that belongs in any budget: Kata needs KVM. Nodes run it fine on bare metal and on clouds with nested virtualization, but a cluster of VMs without nested virt cannot run Kata at all. Confirm that before promising sandboxes on any given node pool.

The decision rule: sandbox the untrusted, share the trusted​

With both prices on the table, the policy writes itself:

• Agent-generated code, build pipelines, anything executing model output → Kata. The workload is untrusted by construction; a second of startup and tens of megabytes are the cheapest breach you will ever buy. Warm pools (which the SIG Apps Sandbox project is building natively) absorb the cold-start cost for latency-sensitive agent loops.
• Trusted tenant production apps → runc. Your own reviewed services gain nothing from a private kernel per pod; spend the overhead budget on more replicas instead.
• GPU inference and confidential data → Kata plus Confidential Containers. Kata 4.0 carried full NVIDIA GPU support into the Rust runtime, and the Confidential Containers stack built on Kata encrypts data in use — the path for agent workloads that process private data on shared accelerators.
• Per-tenant control planes → when tenants are mutually untrusted administratively, i.e. each needs cluster-admin-like freedom without seeing the host. If tenants only need namespace-scoped deploys, soft multi-tenancy plus Kata for the risky pods gets you most of the protection at a fraction of the operational cost.

The through-line: match the boundary to the trust level, per workload, not per cluster. A single fleet can run all three tiers at once — that is precisely what RuntimeClass is for.

What this means for a self-hosted PaaS​

For a platform like Bex.co — Cluster API managing a fleet of owned machines, tenants arriving via git push — Apple's recipe maps almost one-to-one onto the architecture. Tenant environments become virtual control planes provisioned by the same declarative machinery as everything else; agent sandboxes spawned by deploy-from-chat flows land on a Kata RuntimeClass with warm pools hiding the cold start; trusted production services stay on runc; and at no point does untrusted model output share a kernel with a paying tenant's database. The operator checklist is short: enable nested virt or bare-metal KVM on sandbox node pools, install Kata via Kata Deploy, define the RuntimeClass, and provision virtual clusters per tenant through Cluster API instead of namespaces alone. Two layers, two bills, each priced above — and a fleet where the scariest workload on the platform is also the best-contained one.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• Chris Hein & Eric Ernst (Apple), "Running Isolated VirtualClusters With Kata & Cluster API," KubeCon NA 2022 — video / talk listing
• CNCF, "Solving Kubernetes Multi-tenancy Challenges with vCluster" (Sept 2025) — link
• OpenInfra Foundation, Kata Containers 4.0 release (Rust runtime-rs default, agent-sandbox positioning) — release notes / coverage
• Cloud Native Now, "Rust Rewrite Readies Kata Containers for Agent Sandboxing" — link
• Kubernetes SIG Apps, Agent Sandbox project (Sandbox CRD, Kata/gVisor runtimes) — repo
• Community runtime benchmarks: container-runtime-benchmarks (runc vs Kata CLH vs gVisor), micro-containers (cold-start ~20 ms runc to ~500 ms Kata/QEMU), opensandbox cold/warm-start table (Kata QEMU ~1000/200 ms, Firecracker ~625/125 ms), agent-sandbox p50/p95/p99 startup figures
• Loft Labs, vCluster multi-tenancy guides (soft vs hard tenancy, 40M+ virtual clusters) — best-practices guide