Your API Server Still Proxies Every kubectl exec in 1.36: What ExtendWebSocketsToKubelet Really Moves and the 4 Checks Before You Upgrade

Kubernetes 1.36 flipped ExtendWebSocketsToKubelet to beta, and with it came a plausible-sounding scare story for platform teams: exec, attach, and port-forward now flow "straight to the kubelet," bypassing the API server — and taking your central audit trail and policy checkpoint with them. If you run a self-hosted PaaS with a tenant-facing debug shell, that framing demands an urgent redesign. Fortunately, it's wrong. The API server never left the request path. What 1.36 actually moved is smaller, more mechanical, and — once you see the real before/after — checkable with a four-item list instead of an architecture review.

Here's the correction up front, because everything else follows from it. The request path is identical in both phases:

Phase 1 (≤1.35):  kubectl ──WS──▶ API server ──SPDY──▶ kubelet ──SPDY──▶ runtime
Phase 2 (1.36+):  kubectl ──WS──▶ API server ──WS──▶  kubelet ──SPDY──▶ runtime

Same hops. Same proxy in the middle. The only leg that changed protocols is API server to kubelet, and the only work that moved is translation:

So the "audit trail lost" half of the scare story evaporates: the API server still authenticates, authorizes, proxies, and audit-logs every exec session exactly as before. The load it sheds — SPDY encode/decode per streaming connection — lands on the kubelets, distributed across the fleet instead of concentrated on the control plane. That's a scalability win, not a visibility loss. But "nothing to do" would be the wrong conclusion too, because the protocol flip has sharp edges in exactly the places a self-hosted fleet owns: the container runtime, RBAC roles written years ago, and version skew during the upgrade itself.

What the gate actually changes​

The change is phase 2 of KEP-4006, the multi-release project to retire SPDY from the streaming path. Phase 1 (WebSocket from kubectl to the API server, default since 1.30 for exec/attach and 1.31 for port-forward) left the API server doing translation work: accept the client's WebSocket, unwrap it, and speak SPDY upstream to the kubelet. That translation cost scales with every concurrent exec, attach, and port-forward in the fleet — and it all lands on the API server, the one component you least want doing per-connection byte shuffling.

Phase 2 pushes that work to the edge. When both ExtendWebSocketsToKubelet and its dependency NodeDeclaredFeatures are enabled, each kubelet advertises the capability in the declaredFeatures field of its Node.Status. The API server's exec/attach/port-forward handlers check that field on the target node: advertised, and the API server becomes a simple pass-through proxy, forwarding the client WebSocket untouched. Not advertised — or either gate off — and the API server falls back to phase-1 behavior, translating locally and speaking SPDY upstream. On the kubelet side, a StreamTranslatorProxy terminates WebSocket exec/attach sessions and a StreamTunnelingProxy terminates port-forward tunnels, each converting back to SPDY for the final leg to the container runtime. SPDY survives only where it was always hardest to kill: the kubelet-to-CRI boundary.

Three properties of this design matter for upgrade planning. First, negotiation is per-node, so a fleet mid-upgrade degrades gracefully rather than breaking: a new API server talking to an old kubelet just does the translation itself, and an old API server talking to a new kubelet speaks SPDY the kubelet still accepts. Second, the feature is stateless — disable the gate on either component and previous behavior returns with a restart, no migration, no stored state to unwind. Third, nothing about authorization moved: RBAC on pods/exec, pods/attach, and pods/portforward is still evaluated at the API server before a byte is proxied. If your debug-shell access broker grants tenants exec through API-server RBAC, its enforcement point is exactly where it always was.

The policy story that actually matters​

Here's the irony: the RBAC change that did reshape exec authorization already shipped — one release earlier, under a different gate, and it broke real tooling. AuthorizePodWebsocketUpgradeCreatePermission, default-on since 1.35, requires the create verb on pods/exec, pods/attach, and pods/portforward for WebSocket requests too. Previously, SPDY exec required create but WebSocket exec required only get, because the API server derived the RBAC verb from the HTTP method — and a WebSocket handshake is, per RFC 6455, an HTTP GET.

That verb derivation was a genuine authorization hole wearing a protocol detail as a disguise. Roles that granted only get on the exec subresource — wrong in spirit from day one — silently worked for every WebSocket client, which by 1.35 meant nearly everyone. When 1.35 closed it, the breakage surfaced immediately: Gardener's terminal controller filed upstream about attach sessions going permission-denied, and operators found workload automation whose first kubectl exec died with is forbidden because its Role had never granted create. If you upgraded through 1.35 without auditing exec roles, you may be carrying get-only grants that are already broken — and you'll misattribute the failure to 1.36's WebSocket changes if you don't know the two apart.

The audit side needs the same demystification. What the API-server audit log captures for an exec session is the request metadata: who, when, which pod and container, and — because kubectl passes it as a query parameter — the command itself in the requestURI. What it has never captured is the session content: stdin, stdout, and the interactive bytes were never in the audit event under SPDY and aren't under WebSocket either. The gate changes neither half of that sentence. Teams imagining they need "node-level audit logging" to replace something the API server stopped recording are solving a problem 1.36 didn't create. If you want session recording — full transcripts of tenant debug shells — that's a feature your access broker has to build (or buy) at the proxy layer regardless of release, because Kubernetes has never provided it at any layer.

One adjacent hardening note, since debug-shell features invite it: the kubelet's own :10250 endpoints authorize direct WebSocket exec off the same GET-handshake semantics, which is why nodes/proxy is effectively remote code execution and direct-to-kubelet access from tenant networks should stay blocked. That predates this gate and survives it. The gate doesn't open a new direct path — clients still can't reach the kubelet's WebSocket handlers except through the API server's proxy — but every exec-related upgrade is a good excuse to re-verify that no tenant workload can dial a kubelet directly.

The 4 checks before you upgrade​

With the architecture straight, the upgrade work compresses to four concrete checks. Run them in order; each gates the next.

1. Inventory your container runtimes — cri-dockerd breaks. This is the one hard failure 1.36 shipped, and it has already bitten real projects. Kubelets on cri-dockerd don't speak the WebSocket streaming protocol, so exec fails outright once the API server starts proxying WebSocket upstream. Docker Desktop tracked it as a kubectl exec regression on 1.36.1, minikube responded by disabling the gate for the Docker runtime while bumping its default to 1.36, and ceph-csi's CI plus kubespray hit the same wall. containerd and CRI-O are unaffected. If your CAPI node images run containerd — as most Hetzner-based fleets do — you're clear; if any pool still runs cri-dockerd, either migrate the runtime first or pin ExtendWebSocketsToKubelet=false on the API server until you do. This check alone is the difference between a quiet upgrade and a fleet-wide debug outage.

2. Audit RBAC for get-only exec grants. Search every Role and ClusterRole in the fleet for pods/exec, pods/attach, and pods/portforward paired with get but not create — tenant debug roles, CI service accounts, dashboard backends, and especially automation written before 1.35. Anything get-only is already denied on 1.35+ regardless of transport, so fix it as a 1.35 remediation with a 1.36 deadline: grant create where exec is legitimate, and treat any grant you can't justify as a finding, not a formality. While you're there, confirm no tenant-facing identity holds nodes/proxy, which bypasses pod-level scoping entirely.

3. Plan the skew window, then trust the fallback. During a staged rollout, 1.36 API servers will serve a mix of upgraded and not-yet-upgraded kubelets. Verify the behavior you should see rather than assuming it: exec against old nodes keeps working via API-server-side translation (watch for the translation CPU staying on the control plane until the last node drains), and exec against new nodes takes the end-to-end WebSocket path. If your monitoring can distinguish the two — API-server goroutine and CPU profile during a debug-heavy window is the pragmatic signal — you get a free progress metric for the rollout. Node rollouts that replace rather than upgrade in place make this check trivially observable per pool.

4. Know the rollback flag and its blast radius. If anything in checks 1–3 misbehaves, --feature-gates=ExtendWebSocketsToKubelet=false on the API server restores phase-1 behavior cluster-wide with a restart — translation returns to the control plane, SPDY returns to the upstream leg, and clients notice nothing. The KEP explicitly calls out that the feature holds no state, so rollback is safe to exercise and safe to repeat. Set it, fix forward on the actual problem (usually the runtime), and re-enable. Document the flag in the runbook next to the runtime inventory from check 1 so the on-call engineer at 3am doesn't have to rediscover this post.

No redesign, but run the checklist​

The through-line of 1.36's streaming change is reassuringly boring for platform builders: your debug-shell feature needs no architectural change, your audit trail loses nothing, and your policy checkpoint hasn't moved. The API server still stands in the middle of every exec session, authenticating, authorizing, proxying, and logging — it just stopped translating protocols along the way, and your kubelets picked up the work. The real upgrade risks were never in the proxy path; they're in a Docker-shim runtime that can't speak the new protocol, RBAC roles that were quietly wrong since the WebSocket era began, and the usual skew discipline of a fleet upgrade.

That ratio — small mechanism change, checklist-shaped consequences — is exactly why self-hosted fleets should track Kubernetes minor releases as operations inputs rather than feature announcements. The headline says "WebSockets to the kubelet." The runbook says: check the runtime, fix the roles, watch the skew, know the flag. Four checks, no redesign, and your tenants' debug shells keep working through the whole upgrade without ever knowing the protocol changed underneath them.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.