Dora NodaEvery PaaS ships a shell button. Render has a Shell tab that drops you into a running service instance. Heroku has ps:exec, an SSH session straight into a dyno. One click and a developer is inside a live production container, poking at the exact process serving traffic. It is the most loved button on the dashboard and the most dangerous one — because everything that makes it useful to the tenant who owns the workload makes it useful to anyone who can reach one step further.
Here is the hardened pattern up front, before a single paragraph of background. If you run multi-tenant workloads on Kubernetes and you offer anything resembling "shell into your service," this is the whole article in one table:
| Control | What you do | Failure mode it prevents |
|---|---|---|
Debug with ephemeral containers, not exec | kubectl debug --target, never bake shells into prod images | Debug tools doubling as attacker tooling in every tenant image |
| Namespace-scoped RBAC, never cluster-wide | Role permitting pods/ephemeralcontainers in the tenant's namespace only | One tenant's debug session reaching another tenant's pods |
| Non-root by default | runAsNonRoot, drop the sysadmin profile for tenants | A debug container escaping to the node via a privileged profile |
| No node debugging for tenants | Deny kubectl debug node beyond the platform team | chroot /host turning a shell button into root on shared hardware |
| Audit-log every debug action | Log ephemeralcontainers/patch, exec, and attach at RequestResponse | A 2 a.m. "quick look" that nobody can reconstruct afterward |
| Allowlist debug images | Admission policy: only signed internal debug images | A tenant (or intruder) mounting netshoot with a crypto miner inside |
| Time-box and alert | Alert on every ephemeral-container creation; re-audit sessions over minutes old | Debug containers quietly becoming permanent sidecars |
The rest of this post substantiates every row: why exec-as-root is the wrong primitive, what ephemeral containers actually change, the exact threat model that turns debugging into lateral movement, and the copy-pasteable RBAC, audit, and policy snippets that close it.
The habit this replaces: exec into the pod as root
The ad-hoc pattern is kubectl exec -it my-pod -- sh, running as root, with whatever the image happens to contain. It fails twice: once on security, once on modern images.
The security failure is straightforward. exec inherits the container's privileges, and production containers historically ran as root with a full userland — shell, curl, package manager — because someone might need to debug. That means every prod image carries its own burglary kit, and anyone with pods/exec rights gets an interactive root shell with no record of what they typed beyond what the audit log captures of the API call itself (which, by default, does not include the session contents).
The modern-image failure is newer and more final: it just stops working. Best-practice production images are distroless — no shell, no package manager, no sh to exec into. Run kubectl exec -it my-pod -- sh against a distroless image and Kubernetes answers executable file not found in $PATH. The tighter the image, the deader the old debugging habit. That is not a regression; it is the forcing function that pushed the ecosystem to ephemeral containers, which the Kubernetes documentation now treats as the standard answer for debugging running pods.
So the choice is not between debugging and security. The old tool already broke. The question is what replaces it — and whether the replacement ships with guardrails.
What ephemeral containers actually change
An ephemeral container is injected into an already-running pod without restarting it and without changing the pod's spec on disk. It shares the pod's network namespace and, with --target, its process namespace too, so you can strace or inspect the app's filesystem from beside it rather than from inside it. The application image stays distroless; the debugging tools live in a separate, throwaway image that vanishes with the pod.
Three modes cover nearly every case:
- Targeted attach (
kubectl debug pod --target=app --image=debugger:latest -it): joins the pod's namespaces to inspect a live container. The app image needs no shell. - Copy-to (
kubectl debug pod --copy-to=pod-debug --set-image=\*=debug-image): clones the pod for crash-loop or config investigation, leaving the original untouched. The clone runs the debug image in place of the app. - Node debugging (
kubectl debug node/worker-1 -it --image=busybox): drops you onto the node itself, with the host filesystem mounted at/hostand the node's PID, network, and IPC namespaces joined. This is the platform team's tool, never a tenant's.
Since the KEP-1441 work (tracked in sig-cli/1441-kubectl-debug), kubectl debug also ships profiles that bundle capability sets:
| Profile | Powers granted | Who should get it |
|---|---|---|
general | Plain debugging, no extra capabilities | Tenants, on-call engineers |
netadmin | Adds NET_ADMIN and NET_RAW, host namespaces on node debug | Platform networking debugging only |
sysadmin | Broad privileges for deep system inspection | Break-glass platform team, audited |
That table is the whole privilege story in miniature. A tenant shell feature should only ever mint general. Hand a tenant sysadmin on a shared node and you have not shipped a debugging feature; you have shipped a privilege-escalation API with a friendly button.
The threat model: every debug power is a lateral-movement primitive
Security practitioners treat kubectl debug as dual-use, and they are right. The powers below are not exotic exploits — the node-debugging documentation describes them plainly. They are the feature working as designed, pointed at the wrong target:
| Debug power | RBAC that grants it | What goes wrong on shared infrastructure |
|---|---|---|
patch pods/ephemeralcontainers | Lets the holder inject any image into any running pod they can patch | Attacker lands debugging tools (or malware) inside a victim tenant's pod, inheriting its service-account token and secrets |
shareProcessNamespace via --target | Rides along with the debug attach | Sees every process and its environment — including env-var secrets — of the target container |
kubectl debug node | create pods plus node access | Host filesystem at /host; chroot /host is root on the machine, and from the host, every other tenant's containers on that node are readable |
sysadmin / privileged profiles | Profile choice at debug time | Container escape primitives (privileged mode, host namespaces) that dissolve the pod boundary entirely |
Arbitrary --image | No admission control on the debug image | The "debug image" is whatever the caller wants it to be, including images built for exfiltration or mining |
Read that table as an attacker would: one over-broad debug permission plus one shared node equals every tenant on that node. The runC-era container-escape CVEs made the same point from the other direction — once you are on the host, every container on the machine is in scope. Node debugging hands out exactly that position by design, which is why it must never be reachable from a tenant-facing button.
Note the asymmetry that makes this a PaaS problem and not just a Kubernetes problem. On your own cluster, kubectl debug node requires credentials your engineers hold. On a multi-tenant platform, the "user" clicking Shell is a stranger whose workload shares a kernel with other strangers' workloads. The blast radius of a debug feature is not one pod; it is the node, and the node is shared.
The hardened pattern, copy-pasteable
Here is the pattern as enforceable configuration rather than advice. Four pieces, each independently auditable.
1. Namespace-scoped RBAC. The debug role lives in the tenant's namespace and grants exactly three narrow things: reading their own pods, patching ephemeral containers onto them, and creating exec sessions in them. No nodes, no cross-namespace reach, no create pods (which would enable node-debug-style pod creation):
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: tenant-debug
namespace: tenant-acme
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/ephemeralcontainers"]
verbs: ["patch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]Bound with a RoleBinding to the tenant's on-call group — never a ClusterRoleBinding. The moment the binding is cluster-scoped, every row in the threat table above becomes reachable.
2. Non-root, general profile, enforced. The debug container runs as a non-root user, and admission policy rejects anything else. Pair this with restricting tenants to the general debugging profile: no NET_ADMIN, no sysadmin, no privileged flag. The platform team keeps a separate break-glass role for node debugging, bound to humans, not to tenant service accounts.
3. Audit-log every debug action. Ephemeral-container creation, exec, and attach must hit the audit log with request and response bodies, or the session is unreviewable. A minimal audit-policy stanza:
- level: RequestResponse
resources:
- group: ""
resources: ["pods/ephemeralcontainers", "pods/exec", "pods/attach"]Alert on every ephemeral-container creation the way you would alert on a new privileged pod: who created it, against which pod, with which image, and how long it lived. Practitioners who run this in production flag any debug session lasting more than a few minutes for review — a debugger that never goes away is a sidecar, and sidecars deserve the full deployment pipeline, not a debug shortcut.
4. Allowlist debug images at admission. An admission controller (Kyverno, OPA/Gatekeeper) should reject debug containers whose image is not on an explicit allowlist of scanned, signed, internally mirrored images. The rule is one sentence: if the image did not come from our registry, it does not get injected into a running pod. This single control kills the "debug image as malware delivery" row outright, and it costs one policy instead of eternal vigilance.
Ephemeral containers also linger in the pod spec until the pod is deleted, so hygiene matters: the platform should garbage-collect debug sessions aggressively and treat a pod carrying an ephemeral container older than the incident that spawned it as drift to be reconciled away.
What this means for a PaaS shell button
Map the pattern onto the tenant-facing feature Render and Heroku both ship, and the design falls out directly:
- Scope the button to the tenant's namespace and nothing else. The click that opens a shell mints credentials equivalent to the
tenant-debugrole above —generalprofile, non-root, allowlisted image — scoped to that tenant's workloads. There is no code path from "my service's shell" to another tenant's pods or to any node. - Node debugging does not exist for tenants. Not rate-limited, not approval-gated — absent. The platform team debugs nodes with break-glass credentials under the audit policy; tenants never see the option.
- Every session is attributable. The audit trail records which tenant identity opened which shell against which workload, with which image, and for how long. When a tenant asks "who was in our container at 2 a.m.," the answer is a query, not a shrug.
- Distroless-friendly by construction. Because debugging rides in an injected image rather than the app image, tenants get the full shell experience against minimal, shell-less production images. Security posture of the workload and debuggability of the workload stop being a tradeoff.
- Agents get the same treatment. AI operators that deploy and remediate apps will click this button programmatically — and a compromised or confused agent with cluster-wide debug rights is the same lateral-movement story at machine speed. Scope agent debug credentials exactly like human ones: tenant namespace,
generalprofile, allowlisted images, audited.
The uncomfortable corollary: if your shell feature cannot say which of these controls it implements, it is not a hardened feature with gaps — it is the ad-hoc exec-as-root habit with better CSS.
Shells are trust boundaries, not conveniences
The March 2026 Kubernetes guidance landed on a simple thesis: production debugging is a distinct privilege, narrower than admin, wider than read-only, and it deserves its own RBAC shape, its own audit trail, and its own image policy. Ephemeral containers are the mechanism that makes the distinction enforceable — debugging without polluting the app image, scoped without neutering the debugger.
For a self-hosted PaaS, that thesis is the entire shell-feature spec. Tenants sharing nodes is the business model; the debug button is where the tenancy boundary gets tested nightly. Build it as scoped RBAC plus non-root plus audit logging plus image allowlisting, and the button stays a convenience. Skip any one of the four, and it is a lateral-movement path with a friendly label.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
