Retiring cluster-admin and the Shared Bastion: Kubernetes' Access-Broker Pattern for Production Debugging

Here is an uncomfortable fact about the cluster you are probably running right now: Kubernetes audit logs record that someone opened a pods/exec session — who, when, into which pod — but they do not record a single command typed inside it. Exec is bidirectional streaming, and streaming traffic never passes through the audit pipeline's request/response model. If your incident-response plan for "an engineer ran something destructive in prod" is "check the audit log," your plan is a blank page.

The Kubernetes project finally said the quiet part out loud. Its March 18, 2026 guidance on securing production debugging names the two default-bad patterns most self-hosted clusters still run — blanket cluster-admin exec access, and shared SSH bastions with no per-session accountability — and replaces them with an access broker layered on top of RBAC. The architecture stacks three layers, each answering a question RBAC alone cannot:

This post walks through all three layers — including the actual RBAC role from the guidance — and then does the part the guidance leaves as an exercise: wiring the pattern into a multi-tenant self-hosted PaaS's own "give a tenant's developer a debug shell" feature, without that feature quietly becoming the same cluster-admin-by-default anti-pattern with extra steps.

Why the Defaults Fail: Two Escalation Chains and an Audit Gap​

Start with what "just give on-call cluster-admin" actually hands out. Kubernetes' own RBAC good practices read like a map of the blast radius:

• Exec is namespace takeover. A shell in any pod is a shell as that pod's ServiceAccount, with its mounted token and every Secret the workload can read. With list or watch on secrets, kubectl get secrets -A -o yaml dumps every credential in the cluster — get isn't even required.
• Workload creation is node takeover. Anyone who can create pods can schedule a privileged pod with a hostPath mount and own the node's filesystem — which is why the guidance pairs any debug role with Pod Security Standards at Baseline or Restricted.
• nodes/proxy is audit bypass. Permission on nodes/proxy lets a user talk to the kubelet API directly and execute commands in any pod on the node — without passing through API-server audit logging or admission control at all. Even get on that resource is not read-only.

The shared bastion fails differently but just as completely. Shared credentials make attribution impossible: five engineers hold the same SSH key, so "who ran the migration that dropped the table" has no answer. And as the Kubernetes guidance puts it, temporary exceptions become permanent — the emergency access granted during last quarter's incident is still there, because nothing expires it.

Layer the audit gap on top: even a perfectly scoped exec permission produces a log line that says a session happened, never what happened in it. The 2026 Verizon DBIR notes software vulnerabilities (31%) have overtaken stolen credentials as the top initial access vector — but that's a statement about how attackers get in, not a pardon for standing credentials that can't be attributed once they do.

So the defaults fail on all three axes at once: too much may (cluster-admin), unaccountable who (shared keys), and unrecorded what (exec streaming). The broker pattern fixes each with its own layer.

Layer 1: A Least-Privilege Debug Role — the Actual YAML​

The foundation is a namespaced role that grants exactly what an on-call debugging session needs and nothing that escalates. This is the shape the Kubernetes guidance recommends:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: oncall-debug
  namespace: payments-prod
rules:
  # Discover what's running
  - apiGroups: [""]
    resources: ["pods", "events"]
    verbs: ["get", "list", "watch"]
  # Read logs
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]
  # Interactive debugging
  - apiGroups: [""]
    resources: ["pods/exec", "pods/portforward"]
    verbs: ["create"]
  # Understand rollout/controller state
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch"]

Three properties do the security work:

• It's a Role, not a ClusterRole. Scope is one namespace. On-call for payments debugs payments, not the identity service next door.
• No secrets access, no pod creation. Both escalation chains from the previous section are cut at the source: nothing here reads a Secret object or schedules a workload.
• No wildcards. Wildcard grants silently absorb every resource type future Kubernetes versions add; explicit lists don't.

Layer 2: Bind Groups, Never Users​

The guidance's second rule is easy to state and constantly violated: never grant permissions directly to individual users. Bind the role to a group, and let your identity provider — or the broker itself — control who is in the group:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: oncall-debug
  namespace: payments-prod
subjects:
  - kind: Group
    name: oncall-payments
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: oncall-debug
  apiGroup: rbac.authorization.k8s.io

This inversion is what makes temporary access mechanically cheap. Granting an engineer two hours of prod access is now a group-membership toggle in the IdP — an operation that's fast, reversible, and independently logged — instead of a kubectl edit rolebinding against production RBAC that someone must remember to revert. The RBAC objects themselves become stable, reviewable infrastructure that changes via pull request, not via incident adrenaline.

One hard rule from the good-practices doc: never put anyone in system:masters. That group bypasses RBAC and authorization webhooks entirely — no broker can mediate what the authorizer never sees.

Layer 3: The Just-in-Time Gateway — Making Temporary Actually Temporary​

Layers 1 and 2 bound what and who. The third layer is the broker proper: a just-in-time secure shell gateway, typically deployed as an on-demand pod in the cluster, acting — in the guidance's words — as an SSH-style "front door" that makes temporary access actually temporary.

The gateway adds the four controls RBAC has no vocabulary for:

1. Short-lived, identity-bound credentials. No standing kubeconfigs with cluster certs. The broker mints a credential tied to the requester's SSO identity, valid for the session, expiring automatically.
2. Command-level restriction. RBAC can allow pods/exec; only a session-mediating gateway can constrain which commands run inside it — allowing cat and curl while refusing a destructive migration.
3. Approval workflows. Routine requests (logs in staging) auto-approve; sensitive ones (exec in payments-prod) page a human. The policy lives in version-controlled files maintained through pull requests, like any production change.
4. A dual audit trail. The Kubernetes audit log still records the API call; the gateway records the session — closing exactly the gap this post opened with. "Who accessed what, when, and what did they run" finally has a complete answer.

On build-vs-buy: this is a solved category off the shelf — Teleport, for instance, implements precisely this shape for Kubernetes, with SSO-issued short-lived credentials, RBAC-integrated resource scoping, and recorded kubectl sessions. The walkthrough in the PaaS section below is broker-agnostic: it assumes only the three capabilities above (mint expiring credentials, toggle group membership, record sessions), whether they come from an off-the-shelf broker or a minimal one you operate yourself.

Shrink the Role Further: kubectl debug Instead of Exec​

Before wiring the pattern into a platform, note that the newest debugging primitive lets you shrink Layer 1 further. If your production images are distroless — no shell, no ps, no curl, which is exactly how they should be — kubectl exec is useless anyway. kubectl debug attaches an ephemeral container with a real toolbox image to the running pod (GA since v1.23):

kubectl debug payments-api-7d4b9 -it --image=busybox --target=payments-api

The RBAC surface is narrower and different: the debug role grants create on pods/ephemeralcontainers instead of pods/exec. That buys three reductions in blast radius:

• The debug session runs in its own container, not inside the production process's filesystem and environment.
• Production images can stay shell-less permanently — the tooling arrives with the debugger and leaves with the session.
• The permission is distinguishable in policy: your broker can auto-approve ephemeral-container attach while requiring human approval for true exec, because they are different RBAC resources.

The PaaS Test: A Tenant Debug Shell That Isn't cluster-admin in a Trench Coat​

Now the part the guidance leaves as an exercise, and the reason this matters doubly for platform builders. A self-hosted PaaS — one platform team's cluster running many teams' apps in per-tenant namespaces — eventually ships a "debug shell" button: a tenant's developer clicks it and gets a shell in their crashing container.

Here is the trap. The platform's controller already holds a powerful ServiceAccount — it creates namespaces, deploys workloads, manages ingress. The path of least resistance is to let that controller exec into the tenant's pod on the developer's behalf and pipe the terminal through the dashboard. Congratulations: you have rebuilt blanket-cluster-admin exec with extra steps. Every tenant shell now runs as the platform's god-credential, every session is attributed to system:serviceaccount:platform:controller, and your audit trail is one identity doing everything. This is the same anti-pattern the March guidance warns about, wearing your product's UI.

The broker pattern maps onto the feature almost mechanically. A compliant request flow:

1. Developer clicks "Debug shell" on their app. The dashboard sends the request to the broker with the developer's SSO identity — never the platform's.
2. Policy check. The broker evaluates version-controlled policy: is this identity a member of this tenant? Is the target namespace theirs? Staging auto-approves; production may page the tenant's admin.
3. Group toggle, with TTL. On approval, the broker adds the developer to tenant-acme-debug — bound via a per-tenant RoleBinding to a namespaced role shaped exactly like oncall-debug above (swap payments-prod for the tenant's namespace, and prefer pods/ephemeralcontainers over pods/exec). Membership expires automatically at, say, 60 minutes.
4. Credential mint. The broker issues a short-lived credential bound to the developer's identity and scoped to the tenant namespace. No kubeconfig ever lands on the laptop with standing access.
5. Gateway pod on demand. A gateway pod spins up in (or scoped to) the tenant's namespace, mediates the terminal stream, enforces command policy, and records the session.
6. Expiry and evidence. Session ends or TTL fires; group membership and credential evaporate; two artifacts remain — the Kubernetes audit event (this identity created pods/ephemeralcontainers in this namespace at this time) and the gateway's session recording (what they ran).

Note what the platform's own ServiceAccount did in that flow: nothing. It never touches tenant pods for debugging. The developer's identity travels end-to-end, which is the property that makes multi-tenancy auditable at all.

The same flow is why this pattern is becoming urgent rather than merely hygienic: increasingly the "developer" clicking that button is an AI agent operating the deployment. An agent with a standing god-credential is the cluster-admin anti-pattern at machine speed; an agent forced through steps 1–6 gets the same scoped, expiring, recorded access as a human — which is exactly the model AI-native platforms like Bex.co, the open-source Render alternative that treats agents as first-class operators, are built around.

Debugging Access Becomes Ephemeral by Default​

The arc here mirrors what happened to servers a decade ago. We stopped hand-patching pets and made infrastructure immutable; the March 2026 guidance applies the same move to access. Standing credentials are the new snowflake servers: unaccountable, drifting, and one incident away from being someone's initial access vector.

The three-layer stack — least-privilege namespaced roles, group-based bindings toggled by an IdP, and a JIT gateway that mints expiring credentials and records sessions — turns "who can debug production" from a permanent state into a short-lived, approved, evidenced event. For platform teams, the bar is concrete: if your debug-shell feature can't answer "who ran what, in which tenant, approved by whom, expiring when," it isn't a feature yet — it's a breach report waiting for its timestamp.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with AI agents as first-class operators. Star the repo on GitHub or deploy your first app today.