Skip to main content

Your Monitoring Stack Was a Root Shell: What Kubernetes v1.36's Kubelet Authorization GA Actually Fixes

7 min readDora NodaDora Noda
Share
On this page

A ServiceAccount with nothing but nodes/proxy GET — the permission bolted onto practically every Prometheus, Datadog, and cAdvisor sidecar in production — can open a raw WebSocket session against the kubelet on port 10250 and run commands as root in any container on that node. Not read metrics. Not tail logs. Root. Security researchers demonstrated it with a single line:

bash
websocat --insecure \
  --header "Authorization: Bearer $TOKEN" \
  --protocol v4.channel.k8s.io \
  "wss://$NODE_IP:10250/exec/default/nginx/nginx?output=1&error=1&command=id"
 
# uid=0(root) gid=0(root) groups=0(root)

The bug isn't a typo in someone's Helm chart. It's how the kubelet has always authorized WebSocket connections: it checks the HTTP verb on the initial handshake — always GET, because that's how WebSocket upgrades work — instead of the verb the request actually performs. A "read-only" monitoring grant and an exec session hit the same authorization check. One security scan found the pattern in 69 publicly listed Helm charts, including Prometheus, Datadog, and Grafana's own agents. The Kubernetes security team's verdict: working as intended, no CVE — because the real fix wasn't a patch, it was removing the reason anyone needed nodes/proxy in the first place.

That fix — fine-grained kubelet API authorization (KEP-2862) — graduated to GA in Kubernetes v1.36, released April 22, 2026. It's a genuine, overdue improvement. It is also not the fix a lot of people will assume it is once they hear "least privilege." Here's exactly what changed, what it closes, and — more importantly for anyone running a shared-node platform — what it very deliberately does not.

What v1.36 Actually Changes

Before this feature, every kubelet HTTP endpoint/stats, /metrics, /logs, /pods, /healthz, /exec, all of it — funneled through one RBAC subresource: nodes/proxy. Grant it for metrics scraping, and you'd also granted exec. There was no way to ask for one without the other.

KubeletFineGrainedAuthz splits that single chokepoint into distinct subresources per endpoint family:

Kubelet APISubresource
/stats/*nodes/stats
/metrics/*nodes/metrics
/logs/*nodes/log
/pods, /runningPods/nodes/pods
/healthznodes/healthz
/configznodes/configz
/spec/*nodes/spec
/checkpoint/*nodes/checkpoint
/execnodes/exec

The RBAC that used to be one broad grant is now several narrow ones:

yaml
# Before: this ClusterRole can also exec into any container on the node.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-agent-old
rules:
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["get"]
yaml
# After: this ClusterRole can read metrics. That's it.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-agent
rules:
- apiGroups: [""]
  resources: ["nodes/metrics", "nodes/stats"]
  verbs: ["get"]
yaml
# A log collector doesn't need exec either — or metrics access.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: log-collector
rules:
- apiGroups: [""]
  resources: ["nodes/log"]
  verbs: ["get"]
yaml
# Exec is now its own grant, and — this is the important part — it requires
# BOTH verbs the WebSocket handshake conflated before.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: exec-allowed
rules:
- apiGroups: [""]
  resources: ["nodes/exec"]
  verbs: ["get", "create"]

That last block is the actual patch for the WebSocket bug — but only for adopters. The feature shipped through the standard maturity path (alpha and opt-in in v1.32, beta and on-by-default in v1.33, GA and locked-on in v1.36), and the kubelet runs a dual check on every request: try the fine-grained subresource first, fall back to nodes/proxy if that's what the caller still holds. That fallback is why upgrading to v1.36 breaks nothing — and it's also the catch. nodes/proxy itself isn't patched. A ServiceAccount that still holds it is exactly as exploitable via the websocat one-liner above as it was the day this bug was found. GA gives you a narrower grant to migrate to; it doesn't retroactively de-fang the one you're already holding. "No immediate action required" is true and also undersells the actual homework: audit every nodes/proxy grant in the cluster, because that's not a compliance checkbox, it's a still-live exec path.

The Catch: This Is Node-Scoped, Not Tenant-Scoped

Here's where a shared-node platform needs to read the fine print, not the changelog headline. KEP-2862's Non-Goals section says it outright: "Make the kubelet API node-restricted" is explicitly not a goal of this work.

What that means concretely: authorization is scoped by what kind of operation — logs vs. metrics vs. exec — never by which pod. A nodes/log or nodes/exec grant is checked against a Node object, which is cluster-scoped. There is no namespace, no pod name, no tenant boundary anywhere in the RBAC rule. Grant a support tool nodes/exec so it can debug one customer's crashing container, and that grant is good for exec into every other pod colocated on that node too — regardless of which team, which app, or which customer owns them.

On a single-tenant node this distinction is academic. On a multi-tenant PaaS, where bin-packing several unrelated customers' workloads onto the same box is the entire economic point of running a shared platform, it's the whole ballgame. Kubelet-level fine-graining answers "logs or exec?" It does not — and was never designed to — answer "whose pod?"

Where Tenant-Scoped Break-Glass Actually Lives

The good news: the mechanism that does scope by pod and namespace already exists, and it has nothing to do with this KEP. The kube-apiserver's own pods/exec and pods/log RBAC resources — the ones behind kubectl exec and kubectl logs — are namespaced and name-scoped by design, and they were unaffected by the nodes/proxy bug because they never touch the kubelet's raw HTTP API directly; the apiserver mediates and re-authorizes per request.

yaml
# Namespace-scoped: an on-call engineer (or an agent) can exec into pods
# in ONE tenant's namespace. Nothing outside it, on this node or any other.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: tenant-debug
  namespace: customer-acme-prod
rules:
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]

A shared-node PaaS's break-glass story for a support engineer, or an AI agent debugging a tenant's failing deploy, should run through apiserver-mediated, namespace-scoped pods/exec/pods/log — a mechanism that's been GA for years — never through kubelet-level nodes/exec, fine-grained or not. That grant belongs to a much smaller set of actors: genuinely node-level system components — a metrics shipper, a log aggregator, a health-check controller — that legitimately need visibility across every pod on a box, not into one customer's namespace.

And that's exactly the case v1.36 improves. Before GA, giving a node-level observability agent real visibility meant deciding between skipping the feature or accepting nodes/proxy's all-or-nothing exec risk as the cost of admission — the bind that pushed platform teams to build their own proxying and scraping workarounds rather than touch the kubelet API directly. After GA, that agent can hold nodes/metrics and nodes/log and nothing else. It still can't exec into anything, on this node or any other. That's a real, narrow, correctly-scoped win — it just isn't the tenant-isolation win the "fine-grained" name might suggest to someone skimming the release notes.

How This Maps Onto Bex's Agent-Operated Model

Bex's MCP server already treats an AI agent as a first-class caller rather than something that gets dropped into a shell: deploy, restart, get_service, and list_logs are API calls scoped to the App the agent is authorized against, not raw kubectl or kubelet reach — the same namespace-scoped, apiserver-mediated pattern above, by construction. An agent debugging a customer's app talks to that app's logs API; it was never routed anywhere near a node's kubelet endpoint to begin with.

Exec-into-container as an agent tool is a natural next step on that same API surface, and when it ships, this is exactly the design constraint it has to respect: scoped to one App's namespace, never to a shared node. Where v1.36 earns its keep on Bex's side of the stack is upstream of any tenant — the node-level health and log-shipping agents that watch a shared box across every tenant's pods can now hold exactly the metrics and log access they need, and nothing that lets them exec into a single one of those containers.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with agents operating through a scoped API instead of raw cluster access. Star the repo on GitHub or deploy your first app today.


Sources

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex