Skip to main content

Kubernetes Metrics API Is Stable in v1.37: What a Self-Hosted PaaS Must Expose Before Agents Can Safely Autoscale Apps

10 min readDora NodaDora Noda
Share
On this page

Kubernetes 1.37 just did something that sounds like paperwork: it promoted metrics.k8s.io from v1beta1 to a stable v1, nine years after the API first shipped. Nothing about the payload changed. NodeMetrics and PodMetrics look the same, kubectl top still works, and the Horizontal Pod Autoscaler still reads CPU and memory the same way it always did.

The reason this release matters has nothing to do with the API surface and everything to do with who's about to start reading it. A human staring at a Grafana panel can tolerate a metric that's fifteen seconds stale, or a node that silently dropped out of the sample. An agent that calls the metrics endpoint, decides a service is idle, and issues a scale-down cannot. Stability makes the API a safer thing to build automation against — it does not make the automation safe. Before an autonomous agent gets write access to replica counts on a self-hosted platform, the platform owes it four guarantees the API alone doesn't provide: a hard isolation boundary on what the agent can see, an explicit check for stale data, a bound on what a single decision can do, and a machine-readable answer distinct from the dashboard's human-readable one. This post works through what changed in 1.37, why each of those four guarantees is necessary, and what the contract looks like concretely on a Cluster API-provisioned fleet.

What v1.37 Actually Shipped

metrics.k8s.io is the API behind two things every Kubernetes operator already uses: kubectl top and the resource-metrics path of the Horizontal Pod Autoscaler. Under the hood, metrics-server polls every kubelet's Summary API on a fixed interval, aggregates CPU and memory usage per pod and per node, and serves the result as NodeMetrics and PodMetrics objects — the latter broken down per-container.

KEP-5207, led by SIG Instrumentation, graduates that API to Stable v1 in Kubernetes v1.37 "Garhwal", released August 26, 2026. The new v1 surface is structurally identical to v1beta1 — same fields, same semantics, no functional changes. v1beta1 stays usable under Kubernetes's normal deprecation policy, and the Kubernetes project's own release notes say future releases will move kubectl top and the HPA controller over to v1 — meaning in 1.37 itself, the pieces that consume this API haven't all switched yet. The graduation is a stability commitment on the wire format, not a rewrite of what reads it.

That distinction is the whole story for agent-ops. A stable API is a promise that the shape of the data won't shift under you. It is not a promise about freshness, scope, or blast radius — three properties an autonomous consumer cares about far more than a human does, because a human can pause and ask "does this look right?" before acting on a number. An agent that's wired to change replica counts doesn't get that pause unless the platform builds one in.

Why "Stable" Isn't the Same as "Actionable"

Metrics-server has no memory. It polls each kubelet's /metrics/resource endpoint on an interval — 15 seconds by default — holds only the latest reading per pod and node in memory, and discards everything older. There's no time series, no smoothing, no built-in signal for "this number might not reflect current reality." The window field on a PodMetrics object tells you the sampling interval the CPU figure was averaged over, and that's the only staleness hint the API gives you natively.

That gap surfaces in at least three concrete ways, each of which produces a plausible-looking number that means the opposite of what a naive consumer would assume:

A kubelet goes unreachable and metrics-server drops that node's pods from the response — silently. No error, no null field: the pods that were on that node simply aren't in the list. A human glancing at a dashboard notices a chart's shape looks off. An agent that queries the API, sees a smaller pod count than expected, and scales based on the visible fraction would over-scale the pods it can see, compensating for load that was never actually gone.

Metrics-server restarts and serves from a cold cache. The first poll cycle after a restart returns sparse or zero readings for pods it hasn't scraped yet, not because those pods are idle but because the cache hasn't filled. An agent that reads a zero and concludes "this service has no traffic, scale to zero" is acting on an empty cache, not empty demand.

A pod just started, and its window is shorter than the agent assumes. A burst pod sampled seconds after boot can show CPU usage that looks proportionally high for a short window, even though absolute usage is low — the classic small-denominator problem. An agent that reads usage / window without checking window itself would treat a cold-start blip as a sustained spike and scale up for load that was never sustained.

None of these is a bug in metrics-server. They're properties of a system built for a human loop — glance, judge, act — being read by something with no glance step. The fix isn't a different API; it's a rule the consuming agent is required to apply: read window and reject or discount a sample outside an expected range, and treat a pod's absence from the response as "unknown," never as "zero load."

The Isolation Boundary: What Cluster API Actually Buys You

The staleness problem is about when to trust a number. The next problem is about whose numbers the agent is even allowed to see, and this is where a Cluster API-provisioned fleet has a real answer that a single shared cluster doesn't.

In a single Kubernetes cluster serving multiple tenants, pods.metrics.k8s.io access is scoped with RBAC: a namespaced Role bound to a tenant's ServiceAccount grants get/list on that namespace's pod metrics and nothing else. That works, but it's a permissions boundary inside one shared control plane — one misconfigured ClusterRole, one broadened RoleBinding, and a tenant's autoscaling agent can list another tenant's pod metrics. It's a real boundary, and it needs to be correct, but it's the same kind of boundary that a single bad YAML diff can quietly widen.

A Cluster API-backed PaaS has a stronger option available, because CAPI's whole job is provisioning workload clusters, not just namespaces. Give each tenant — or each pricing tier — its own CAPI-provisioned workload cluster, and a tenant's agent reaching metrics.k8s.io is talking to a distinct kube-apiserver with no code path to another tenant's data at all. There's no RoleBinding to misconfigure across tenants, because there's no shared control plane for a misconfigured binding to leak across. This is the isolation guarantee a namespace can't give you: not "the permission says no," but "the API server has never heard of the other tenant's pods."

That's not a reason to skip RBAC. Dense, cost-sensitive tiers still put multiple tenants in one workload cluster to amortize control-plane overhead, and inside that shared cluster, namespace-scoped Role + RoleBinding on pods.metrics.k8s.io is exactly the fallback boundary described above — necessary, but explicitly secondary to the per-tenant workload-cluster boundary wherever CAPI makes that boundary affordable.

The Admission Limit: Bounding What a Decision Can Do

Isolation and staleness checks govern what an agent can read. The last piece governs what it can do with that read — because even a perfectly fresh, correctly scoped metric can be fed into a bad decision, and a platform that lets any read produce any write has no defense against that.

Kubernetes 1.37 sharpens exactly this failure mode with HPA scale-to-zero (KEP-2021), graduating to Beta and enabled by default in this release. Setting spec.minReplicas: 0 now lets an HPA scale a workload down to zero pods and back up when demand returns — genuinely useful for queue consumers, batch jobs, and idle GPU pools. But the feature comes with an explicit, load-bearing limitation: scale-to-zero is not supported for CPU or memory metrics, only for object or external metrics, because once replicas hit zero there's no running pod left to report CPU or memory usage from. An agent that tries to drive a scale-to-zero decision off a PodMetrics CPU reading isn't just picking a suboptimal signal — it's using a metric the mechanism explicitly doesn't support for that action, which is exactly the class of mistake an admission boundary exists to catch before it reaches the cluster.

Concretely: say a tenant's worker deployment sits at 3 replicas, and the agent reads pods.metrics.k8s.io and sees near-zero CPU across all three for the last two sampling windows. Without a bound, a "helpful" agent submits minReplicas: 0 on a CPU-driven HPA — a change Kubernetes 1.37 will silently decline to honor as scale-to-zero (CPU isn't a supported zero-scale signal), leaving the tenant with a config that looks like it opted into scale-to-zero but never will. An admission layer that caps replica-count deltas per action, requires a cooldown between successive scaling calls on the same workload, and rejects minReplicas: 0 submissions against CPU/memory-backed HPAs specifically would catch that request before it's applied — and log why, so the tenant sees a rejected action instead of a silently inert one.

None of this is Kubernetes's job to enforce — the API happily accepts whatever HorizontalPodAutoscaler spec you give it. It's the platform's job, sitting between the agent and the cluster's write path.

The Contract, Assembled

Put together, a Cluster API-backed PaaS that wants to let an agent read utilization and change replica counts owes it four things before the first automated scaling call, not after the first incident:

SurfaceIsolation boundaryStaleness checkAdmission limit
pods.metrics.k8s.io (dedicated tenant tier)Separate CAPI-provisioned workload cluster per tenant — distinct kube-apiserver, no shared RBAC surfaceReject samples with window outside expected range; treat an absent pod as unknown, never zeroN/A (read-only surface)
pods.metrics.k8s.io (shared tenant tier)Namespace-scoped Role + RoleBinding on pods.metrics.k8s.io, bound to the tenant's ServiceAccountSame window/absence checks, plus explicit handling for a post-restart cold-cache windowN/A (read-only surface)
HPA / replica-count writesScoped to the requesting tenant's own workloads onlyDecision must cite the specific PodMetrics/window it acted on, for auditMax replica delta per action; cooldown between actions on the same workload; reject minReplicas: 0 against CPU/memory-backed HPAs
Dashboard vs. agent read pathSame underlying dataDashboard renders whatever's current, unlabeled; agent path requires the freshness check above before useDashboard is display-only; only the bounded write path can change cluster state

That last row is worth naming directly, because it's easy to build one internal endpoint for both and assume the freshness question is answered once. A human looking at a graph that's a few seconds behind reality isn't at risk — they'll notice if the number looks wrong before they act on it. An agent given the same endpoint and no separate freshness gate will act on whatever the query returns, on schedule, without a "does this look right?" step. The dashboard and the agent read path can share a data source. They cannot share a trust model.

Kubernetes 1.37 didn't create any of these four requirements — RBAC, staleness, admission control, and read/write separation all predate this release. What the metrics.k8s.io graduation does is remove the excuse for skipping them: the API underneath agent-driven autoscaling is now a stable, nine-years-proven contract, which means the remaining gap between "an agent can read this" and "an agent should be allowed to act on this" is entirely the platform's to close.


Bex is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with a Cluster API-provisioned fleet underneath and AI agents as first-class operators rather than an afterthought bolted onto a dashboard. Star the repo on GitHub or deploy your first app today.

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex