Dora NodaMulti-tenancy on Kubernetes has always been a trilemma with two bad corners. Share one cluster and every tenant lives in a namespace, which is cheap until one tenant needs their own CRDs, their own RBAC model, or a different Kubernetes version. Or give every tenant a whole cluster, which is proper isolation at the price of an etcd quorum, a control plane, and a load balancer per tenant — absurd past a dozen tenants. Rancher's K3k project picks the third corner: run a full K3s control plane per tenant, as pods, on one shared host cluster. The tenant gets a kubeconfig that behaves like a real cluster. The operator provisions no new machines.
That is the whole pitch, and the comparison table below is the whole verdict. Everything after it is evidence.
| Shared mode (default) | Virtual mode | Namespaces (status quo) | Cluster per tenant | |
|---|---|---|---|---|
| Tenant's isolation boundary | Own API server, workloads reflected onto host | Own API server + own nested workers | Namespace + RBAC + quota | Entire cluster |
| Marginal control plane per tenant | One agentless K3s server pod | Server pod + ≥1 agent pod | ~Zero | Full HA control plane + machines |
| Tenant workload execution | On host nodes (via virtual kubelet) | On nested K3s agents (host pods) | On shared nodes | On dedicated nodes |
| Networking | Host CNI + NetworkPolicies | Own CNI per virtual cluster | Host CNI + NetworkPolicies | Own everything |
| Storage | Host StorageClasses | Host storage via nested mounts | Host StorageClasses | Own storage |
| Quota model | ResourceQuota + LimitRange on host namespace | CPU/mem limits on the cluster's pods | ResourceQuota per namespace | Cloud bill per cluster |
| Own CRDs / admission / version skew | Yes | Yes | No | Yes |
The honest summary: shared mode is namespaces with an API-server upgrade — nearly the same density, dramatically better tenant autonomy. Virtual mode is a real cluster with a fake floor — full-stack isolation whose nodes are pods, at two to three times the per-tenant control-plane footprint. The rest of this post shows how each mode works under the hood, prices the fleet-scale overhead the table summarizes, and names exactly where the isolation stops — because a virtual control plane is still a tenant-shaped slice of somebody else's kernel.
How K3k actually works: one CRD, two topologies
K3k's operator surface is small. A controller runs on the host cluster watching Cluster custom resources. Creating a Cluster CR provisions a namespace, the K3s server pod(s), networking, and policy bindings; the k3kcli hands back a kubeconfig. Central defaults come from VirtualClusterPolicy objects bound to host namespaces, so an admin sets guardrails once per namespace instead of repeating them per tenant. That is the entire provisioning path — no machine objects, no bootstrap tokens on real metal, no per-tenant load balancer. If you run Cluster API today, notice what disappeared: the whole infrastructure-provider layer. The host cluster is the infrastructure.
Shared mode is the clever one. The virtual cluster's control plane is a K3s server running in agentless configuration: no kubelet, no container runtime, no CNI inside the server pod. In place of agents, K3k registers its own virtual kubelet provider, which reflects the virtual cluster's pods and supporting objects down onto the host cluster, where the host's kubelets actually run them. Each reflected pod gets a unique name derived from pod, namespace, and cluster, so tenants can collide with each other all day without colliding on the host. Scheduling is therefore free — the host scheduler already bin-packs everything — and the tenant's kubectl get pods shows their world exactly as they declared it.
Virtual mode drops the cleverness for brute-force fidelity. Each virtual cluster is a complete nested K3s deployment — dedicated server pod plus one or more agent pods — running inside the host. The nested cluster brings its own CNI, its own scheduling domain, its own everything; from the tenant's perspective there is no host at all. The price is that every level of the stack now exists twice: the host schedules agent pods, and the nested scheduler schedules tenant pods onto those agents.
What an API server per tenant actually buys
Namespace multi-tenancy fails tenants in four specific ways, and a virtual control plane fixes all four. First, CRDs are cluster-scoped: on a shared API server, one tenant's Crossplane composition or Knative install is everybody's API surface, including version conflicts. A virtual cluster gives each tenant their own API surface to extend. Second, RBAC is bounded by what the platform team grants on the shared server; cluster-admin inside your own virtual cluster is a safe thing to hand out, because it administers nothing outside it. Third, admission control — the webhooks that enforce policy — can be tenant-owned instead of platform-imposed. Fourth, version skew: one tenant can sit on an older Kubernetes while the platform moves on, which is impossible when everyone shares an API server.
That fourth point deserves emphasis for a PaaS audience. Forcing fleet-wide Kubernetes upgrades on every tenant simultaneously is one of the sharpest operational edges of namespace tenancy — the tenant with a deprecated-API dependency becomes your upgrade blocker. Virtual clusters convert a coupled fleet upgrade into N independent, tenant-paced upgrades. The cost of that decoupling is the subject of the next section, but the benefit is real: your slowest tenant stops setting the pace for your control plane.
The fleet-scale cost, quantified
Here is the math the trilemma table summarizes. Treat every number as a sizing budget to validate, not a benchmark — measure your own server pods under your own tenant load.
A K3s server wants on the order of 512 MB of RAM before it holds anything — that is the project's own documented baseline, and an agentless server still runs a full API server plus its datastore. Budget roughly 0.5–1 GB RAM and a fractional vCPU per virtual-cluster server pod in shared mode. Virtual mode adds at least one agent pod per tenant (a kubelet plus containerd is another few hundred MB and a scheduling footprint), so figure two to three times shared mode's per-tenant control-plane cost.
Project that to 100 tenants on owned hardware:
- Namespaces: ~zero marginal control-plane RAM. One API server, one etcd.
- K3k shared mode: ~100 server pods, on the order of 50–100 GB of fleet RAM spent on tenant API servers, zero extra nodes, zero extra load balancers.
- K3k virtual mode: ~100 server pods plus 100+ agent pods — low hundreds of GB and meaningfully more scheduling fragmentation, since nested agents are coarse bins.
- Cluster per tenant: 100 etcd quorums, 100 API load-balancer endpoints, 100 machine lifecycles. Nobody does this at 100 tenants, which is exactly the point — K3k exists because this row is disqualifying.
Two conclusions fall out. First, shared mode's overhead is a memory tax, not a machine tax: it fits inside the fleet you already run, which is why density-sensitive platforms default to it. Second, virtual mode's overhead is coarse enough that you should reserve it for tenants whose isolation need justifies whole nested workers — regulated workloads, untrusted control-plane behavior — not as the default for every free-tier signup.
Then there is the second bill: upgrades. N virtual clusters means N control planes to patch, and tenant-paced upgrades cut both ways — your tenants can lag, which means you operate a version matrix instead of a version. K3k's answer is VirtualClusterPolicy: central, namespace-bound policy so version floors, runtime classes, and resource defaults roll out declaratively rather than by editing a hundred Cluster CRs by hand. It mitigates the sprawl; it does not eliminate it. Anyone evaluating virtual clusters should price the upgrade matrix alongside the RAM — the Rafay critique of the virtual-cluster space lands here, that per-cluster add-ons and patch lifecycles are the overhead that survives after the provisioning demo ends. K3k keeps the per-cluster surface minimal (a server pod, not a full addon stack), but at hundreds of tenants the matrix is still the job.
GPU multi-tenancy: the 2026 chapter gets its own section
The newest part of this story — and the one most relevant to agent-sandbox roadmaps — is GPUs. At KubeCon EU 2026 SUSE announced Virtual Cluster GPU Multi-Tenancy via K3k in Rancher Prime: each tenant gets a fully isolated control plane on shared GPU infrastructure, with automated quota management dividing accelerator capacity between tenants. The ClearML partnership from March 2026 is the adoption evidence that this is not slideware — ClearML's AI control plane sits on top of K3k virtual clusters to hand production-ready, tenant-isolated AI environments to enterprise teams while pooling the underlying GPUs.
Read the fine print in two layers, because both are true at once. The product layer (Rancher Prime packaging: per-tenant control plane plus quota management on shared GPU nodes) is shipping and commercially supported. The upstream layer is younger: K3k's own architecture docs still list GPU resource sharing as an area of ongoing investigation — quota-bounded assignment of devices to tenants exists, fine-grained sharing (time-slicing, fractional allocation, MIG-style partitioning through the virtual layer) is still maturing. That gap matters for capacity planning: today K3k GPU tenancy answers "whose GPU is this" more completely than "how do eight tenants share one H100." The trajectory — GA virtual clusters, CNCF conformance listing, a priced GPU-multi-tenancy product, and a major MLOps integration inside one year — is the maturity signal; the sharing granularity is the thing to re-check every quarter before promising tenants fractional GPUs.
For context, the competitive pressure is real: vCluster Labs shipped an Infrastructure Tenancy Platform for AI GPU efficiency in late 2025, and the whole virtual-cluster space has realized that whoever solves tenant-isolated GPU sharing owns the agent-sandbox substrate. K3k's bet — full K3s control planes, host-scheduler bin-packing in shared mode — is the density-maximalist entry in that race.
What this means for a self-hosted PaaS
Map it onto the isolation ladder most self-hosted platforms already climb. Rung one is namespaces plus quotas. Rung two adds sandbox runtimes — gVisor, Kata — so a tenant's compromised container still faces a kernel or VM boundary. K3k is rung three: the tenant stops sharing your API server at all. Each rung costs more per tenant and buys a strictly stronger boundary, and the honest question is which rung each tenant tier needs. Free-tier web apps never need rung three. A team deploying operators, custom CRDs, and admission webhooks — or an AI-agent sandbox tenant whose workload is Kubernetes API calls — outgrows rung two the day they ask for cluster-admin and you have to say no.
The adoption sequence writes itself: shared mode for tenant control planes at density, virtual mode reserved for the tenants whose threat model or compliance scope justifies nested workers, VirtualClusterPolicy keeping the hundred-cluster matrix governable, and GPU tenancy extended exactly as fast as the upstream sharing primitives mature. SUSE putting virtual clusters through CNCF conformance and into a GA product means the primitives are no longer experimental — the remaining work is operational judgment about placement, not bets on whether the technology exists.
K3k is the open-source engine behind SUSE Virtual Clusters — explore the architecture docs or spin up a first virtual cluster with k3kcli against any host cluster you already own. Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
