Kubernetes 1.37 Didn't Graduate CBOR: The Alpha Opt-In That Cuts Custom-Resource Encode Costs 8x

Half the Kubernetes 1.37 roundups will tell you CBOR graduated. One widely shared recap lists the CBOR serializer as stable; another has it graduating to beta. Both are wrong, and you can check in thirty seconds: in the v1.37.0 source tree, the CBORServingAndStorage feature gate is still alpha and still defaults to false. The KEP's own milestone file says beta was the target for 1.37. The code says the flip never landed.

That gap between "announced" and "shipped" is the entire reason to read this post. CBOR for custom resources is real, it is measurable — roughly 8x faster encodes and 2x faster decodes for CR and dynamic-client operations in the KEP's benchmarks — and 1.37 did move it forward in concrete ways. But if you run a control plane whose tenant state lives in CRDs, you need the code truth, not the roundup truth: what the feature actually buys, what 1.37 changed under the hood, and the exact steps to turn it on and back off again. That is what follows.

The verdict up front: still alpha, still worth testing​

Here is the status table, verified against primary sources rather than blog posts:

The practical verdict: do not put this on a tenant-facing default yet — alpha means the API surface can still change and the gate is off for a reason. Do turn it on in a staging management cluster this month, because the opt-in is small, the rollback is clean, and the payoff lands exactly where CRD-heavy control planes hurt. The rest of this post is the evidence and the runbook.

Why custom resources pay a JSON tax no native type pays​

Native Kubernetes types cheat. When kube-apiserver talks to etcd, or when kubectl negotiates a response, built-in objects travel as Protobuf: a schema-driven binary format that skips field names on the wire and leans on generated code both sides already have. It is dramatically cheaper than JSON to marshal and unmarshal, which is why the apiserver prefers it everywhere it can.

Custom resources cannot use Protobuf at all. Protobuf depends on code generation and on both sides holding compilation-time knowledge of the schema — the opposite of what a CRD is. A CRD's schema is declared at runtime, in YAML, by whoever applied it. So every custom object pays the full JSON price on every hop: serialized to JSON for etcd storage, deserialized from JSON on every read, re-serialized to JSON for every list, watch event, and controller reconcile. For a cluster with a handful of CRs this is noise. For a management cluster whose controllers list and watch thousands of Machine, MachineDeployment, and platform-tenant objects in a tight loop, it is a standing tax on apiserver CPU and a heap-churn machine: every decode allocates, every encode allocates, and the garbage collector bills you for all of it.

That is the gap KEP-4222 (authored by Ben Luddy, owned by sig-api-machinery, alpha since 1.32 behind CBORServingAndStorage) exists to close. CBOR — Concise Binary Object Representation, RFC 8949 — is self-describing like JSON but binary: no field-name repetition without a schema, no base64-style bloat, no whitespace, and crucially no codegen requirement. The KEP's motivation benchmarks report custom-resource and dynamic-client encode operations up to ~8x faster and decode operations ~2x faster than JSON, with substantially fewer heap allocations. Treat those numbers as what they are — KEP-era motivation benchmarks for CR and dynamic-client paths, not measurements of your 1.37 apiserver — but the direction is unsurprising: a binary format parsed without building an intermediate generic map will always beat encoding/json on this workload.

The Go implementation rides on fxamacker/cbor/v2 (bumped from v2.9.0 to v2.9.1 in the 1.37 dependency refresh), which the KEP's library evaluation clocked at a 2.4x encode speedup over the alternative, plus another 1.8x available from disabling map-key sorting.

One honest caveat before the runbook: your speedup depends on object shape. Small objects with few fields gain less; large status blobs, wide lists, and high-frequency watch streams gain most. Measure your own apiserver — apiserver_request_duration_seconds broken down by resource, plus apiserver CPU — before and after, rather than budgeting the KEP's headline numbers.

What 1.37 actually shipped for CBOR​

With the beta flip out of the picture, what did Garhwal concretely add? Two changelog entries, both merged behind the still-alpha gate, both aimed at checklist items from the KEP's own beta criteria:

CBOR reached the structured endpoints (PR #139632). Discovery endpoints and structured error responses now support CBOR encoding when the gate is on. This sounds cosmetic until you run a dynamic client: discovery documents are the largest payloads many controllers fetch, and every agent, operator, and GitOps tool that boots against your management cluster downloads them. It also closes a correctness gap — an endpoint that answers errors in JSON while the client negotiated CBOR is an interop bug waiting for a 3 a.m. page.

List responses encode item-by-item (PR #138808, by chenk008). Previously the apiserver's CBOR encoder buffered an entire collection before writing it out, so a large list cost memory proportional to its size. Now collections encode streaming, one item at a time. This is nearly verbatim one of the KEP's beta exit criteria ("collection encoding supports true streaming, i.e. buffer size is not proportional to output size"), which tells you 1.37 was a progress toward beta release for this feature even though the graduation itself slipped.

What remains before beta: the gate flip itself (alpha → beta, default-true), the final decision on keeping the nondeterministic encoding mode, finishing the automatic-transcoding opt-outs for FieldsV1/RawExtension, and per-resource 415 fallback granularity in client-go. None of that blocks you from testing the alpha — it only means you should test it the way you test any alpha: staged, observed, reversible.

The opt-in checklist: server, clients, storage​

Everything below assumes Kubernetes 1.32 or newer on both apiserver and clients, with 1.37 preferred for the two improvements above. Work through the checklist in order; each step is independently reversible.

Step 1 — Enable the gate on every apiserver. The gate lives on kube-apiserver only:

--feature-gates=CBORServingAndStorage=true

Roll it across all apiserver replicas before any client starts asking for CBOR. A client that sends a CBOR body to an apiserver without support gets 415 Unsupported Media Type, and while client-go falls back to JSON on 415, you do not want a mixed-gate fleet during rollout — half your apiservers negotiating one encoding and half another is exactly the kind of skew that produces irreproducible latency graphs.

Step 2 — Point a client at CBOR explicitly. Nothing changes by default when the gate flips on; clients must ask. For raw HTTP that means Content-Type: application/cbor on writes and Accept: application/cbor on reads. In client-go, set the ContentType field of rest.ClientContentConfig — the same knob you would use to prefer Protobuf today. Watches travel as CBOR Sequences (application/cbor-seq), and Server-Side Apply plus strategic merge patches get +cbor media-type suffixes (application/apply-patch+cbor). Two deliberate omissions: JSON Patch and JSON Merge Patch have no CBOR variants, because both are JSON documents by definition and a parallel spec would buy nothing.

Two client-side gates, AllowCBOR and PreferCBOR, control whether configured CBOR preferences are honored or silently rewritten to JSON — the escape hatch if a client misbehaves.

Step 3 — Understand what happens in etcd. Enabling the gate changes the default storage encoding for custom resources to CBOR. This is invisible to clients — the apiserver still serves whatever encoding each client negotiates — but new and rewritten objects land in etcd as CBOR. Existing JSON-encoded objects are not rewritten automatically; mixed storage is fully supported, with the decoder recognizing CBOR by its 0xd9d9f7 magic prefix whether the gate is on or off. To convert the backlog, do a no-change get-and-put of each object, or automate it with the StorageVersionMigrator — the same tool the 1.37 storage-migration GA story covers. Note the asymmetry the KEP is explicit about: built-in types keep their existing storage encoding; only custom resources move.

Step 4 — Mind the skew boundaries. Aggregated API servers negotiate independently — CBOR works at the aggregation layer only if the aggregated server enables it too. And keep kubectl within supported skew: the KEP promises the default request content type stays JSON until at least two minor versions past GA, precisely so an older client never requires CBOR from a newer server. During your alpha test, pin client versions and watch for 415s in apiserver logs as the signal that something in the chain doesn't speak the new encoding yet.

The rollback runbook and the one metric to watch​

Alpha features earn trust by being easy to turn off, and this one is. If anything looks wrong — latency regression, decode errors, a client stuck in a fallback loop — the rollback is two moves:

Move 1 — Disable the gate and restart the apiservers. Set CBORServingAndStorage=false and roll the restart. Clients sending CBOR bodies trip a client-side circuit breaker and fall back to JSON automatically (for the life of the RESTClient in alpha), and operators can force the issue by disabling the client-go ClientsAllowCBOR gate and restarting the client. No flag-day coordination: because every apiserver version that knows about CBOR can decode CBOR-encoded objects from storage whether or not the gate is enabled, a half-rolled fleet still reads everything.

Move 2 — Migrate storage back only if you want to. You do not have to. Mixed JSON/CBOR storage remains fully readable with the gate off, indefinitely. If you prefer a clean etcd — say, before downgrading to a release that predates CBOR decoding entirely — repeat the no-change get-and-put pass (or a StorageVersionMigrator run) with the gate disabled and objects land back in JSON.

The single metric that decides "stay or roll back" is storage_decode_errors_total. The KEP names it explicitly as the rollback signal, and the correct value is zero, always. Any nonzero reading means a persisted object failed to decode — a roundtrip bug, not a performance tradeoff — and you stop the experiment until it is explained. Pair it with the before/after apiserver CPU and request-latency comparison from the previous section, and you have the complete evaluation: did control-plane CPU drop, did p99 for CR list/watch move, and did the decode-error counter stay flat at zero.

One more piece of the safety story worth knowing: storage uses CBOR's deterministic encoding mode (stable byte output, suitable for etcd comparison and hashing), while serving uses the faster nondeterministic mode (map order varies, like Go map iteration — which is deliberate, so nobody accidentally depends on key order). If you ever diff raw etcd values between two writes of the same object, that split is why serving bytes can differ while stored bytes do not.

Why a PaaS management cluster feels this first​

Most workload clusters will barely notice CBOR. Their hot paths are Pods and Services — native types already on Protobuf. The clusters that feel the JSON tax are the ones whose control plane state itself is custom resources. A Cluster API management cluster is the textbook case: every Cluster, Machine, MachineDeployment, and MachineHealthCheck is a CR, watched continuously by controllers that reconcile on every change. Layer a git-push PaaS on top — deploy records, build statuses, per-service route objects, tenant records, all CRDs — and a burst of concurrent tenant deploys becomes a burst of JSON marshal/unmarshal contention on the same apiserver CPUs serving those watches.

That is why this feature, alpha or not, belongs on a self-hosted platform team's staging checklist now rather than in a "wait for GA" backlog. The enablement is one flag, the client migration is one config field per controller, the storage migration reuses tooling 1.37 just graduated to GA, and the rollback is a flag flip plus an optional rewrite pass. When the beta flip does land — the KEP targets it, the 1.37 streaming and discovery work checked off its prerequisites — the teams that already measured their own before/after numbers will know on day one whether to leave it on. The teams that waited on the roundups will still be arguing about whether it shipped.

Kubernetes 1.37 (Garhwal) shipped August 26, 2026 with the CBOR serializer still in alpha — off by default, moving forward, and ready to test. Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• KEP-4222: CBOR Serializer (keps/sig-api-machinery/4222-cbor-serializer), including kep.yaml (alpha v1.32, beta targeted 1.37) — kubernetes/enhancements
• CBORServingAndStorage gate definition, still alpha/default-false in the v1.37.0 tag and on master — kubernetes/kubernetes, kube_features.go
• Kubernetes 1.37 changelog: PR #139632 (CBOR in discovery endpoints and structured errors), PR #138808 (item-by-item collection encoding), fxamacker/cbor v2.9.0 → v2.9.1
• "CBOR resource encoding," kubernetes.io API Concepts: "Feature state: Alpha since Kubernetes v1.32; disabled by default" — kubernetes.io
• Cloudsmith, "Kubernetes 1.37: What You Need to Know" (CBOR "graduating to beta" claim — contradicted by the v1.37.0 source) — cloudsmith.com
• DevOps Daily, "Kubernetes 1.37 Garhwal: What Shipped and What Slipped" (67 enhancements; fact-check note on CBOR graduation claims) — devops-daily.com
• CBOR specification, RFC 8949 — datatracker.ietf.org