Skip to main content

Kubernetes Quietly Fixed a Years-Old cgroup v1-to-v2 CPU Priority Bug: What the January 2026 Formula Rewrite Means

10 min readDora NodaDora Noda
Share

If you've been running Guaranteed-QoS pods on a cgroup v2 node any time since Kubernetes support went GA, a container that requested a full CPU has not been getting a full CPU's worth of priority. It's been getting about 39% of it. Not because you misconfigured anything — because the formula Kubernetes used to translate your CPU request into a kernel scheduling weight was wrong, and it stayed wrong for years without a single feature-gated warning.

The Bug, In One Table

Every CPU request a kubelet hands to a container runtime gets translated twice. First, 1000m becomes 1024 cgroup v1 CPU shares — the old, familiar unit, still emitted by the kubelet regardless of which cgroup version the node actually runs. Then, on a cgroup v2 host, the runtime converts those shares into cpu.weight, the unit the kernel's CFS scheduler actually uses to divide CPU time between sibling cgroups. That second conversion is where things went wrong.

The original formula, from KEP-2254, was a straight line:

text
cpu.weight = (1 + ((cpu.shares - 2) * 9999) / 262142)

It looks harmless — a linear rescale from the v1 shares range [2, 262144] to the v2 weight range [1, 10000]. But cgroup v1 and cgroup v2 don't share a default. v1's default is 1024 shares; v2's default is 100 weight. A correct conversion should map one to the other. This formula doesn't:

CPU requestv1 sharesOld weight (buggy)New weight (fixed)
100m1024~11
250m25610~25
500m51220~50
1000m (1 CPU)102439100
2000m204879~199
4000m4096159~399

A container requesting exactly one CPU — the single most common request in any fleet — landed at cpu.weight = 39. Cgroup v2's own default weight, the one every non-Kubernetes process on the box gets if nobody touches it, is 100. Your pod asked for parity with the host's baseline and the kernel silently gave it 39% of that instead. Every row in the table shows the same undershoot, not just the 1-CPU case: this wasn't a rounding quirk at one data point, it was the shape of the whole formula.

The fix, landed January 30, 2026, replaces the line with a quadratic curve calibrated to pass through three points instead of two:

text
cpu.weight = ceil(10 ^ (L² / 612 + 125L / 612 − 7/34))
where L = log2(cpu.shares)

That curve hits (2, 1) and (262144, 10000) at the boundaries, same as before — but it also hits (1024, 100) in the middle, which the old linear formula couldn't do without breaking one of the endpoints. One CPU now converts to cpu.weight = 102, matching the host default it was always supposed to match.

Why This Sat Unnoticed Since the cgroup v2 Default Landed

cpu.weight isn't a Kubernetes API field. kubectl describe pod shows you requests.cpu. kubectl top and metrics-server show you actual CPU usage. Neither shows you the number the kernel scheduler is actually enforcing when your pod competes for CPU time against a sibling cgroup — a system daemon, a DaemonSet, another tenant's pod on the same node. The weight lives only in a file under /sys/fs/cgroup, written once at container creation by the OCI runtime, and nothing in the normal observability path ever reads it back out.

That's the mechanism behind the "quietly" in this bug's whole story. The scheduler admitted your pod because the request was satisfiable — there was 1 CPU worth of allocatable capacity on the node. The request was honored in the sense Kubernetes cares about: bin-packing, admission, eviction thresholds all worked off the 1000m number correctly. What broke was one layer further down, in a conversion the kubelet doesn't perform and the API server never sees — the container runtime's own translation from shares to weight. A tool built to watch Kubernetes objects has no reason to know that layer exists, let alone that it's been arithmetically wrong since the day the conversion first went into production.

The bug was filed against runc in May 2025 — nearly three years after cgroup v2 support went GA in Kubernetes 1.25, and most Linux distributions had long since switched to it as their own default — by someone who happened to go looking at the raw cgroup file on a real node and noticed the number didn't match what they expected. The parallel kubernetes/kubernetes#131216 thread makes the same point from the other direction: this was found by inspection, not by any dashboard, alert, or release note pointing someone at it.

Check Your Own Nodes Right Now

You don't need to trust a blog post's table. You can read the number the kernel is actually enforcing on any node you operate, in under a minute.

First, find a container's cgroup path via the CRI:

bash
CONTAINER_ID=$(crictl ps --name <your-container-name> -q)
PID=$(crictl inspect "$CONTAINER_ID" | jq -r '.info.pid')
CGROUP_PATH=$(awk -F: '$3 != "" {print $3; exit}' /proc/"$PID"/cgroup)

Then read the weight the kernel actually has on file:

bash
cat /sys/fs/cgroup"$CGROUP_PATH"/cpu.weight

Compare it against the CPU request in that pod's spec, using the table above. If a pod requesting 1000m shows cpu.weight in the high 30s, that node's container runtime is still running the pre-2026 linear formula — full stop, regardless of which Kubernetes minor version the cluster is on, because this conversion never lived in Kubernetes code to begin with.

Which is the second thing worth checking, and the one that actually determines whether upgrading fixes anything:

bash
runc --version   # want 1.3.2+
crun --version   # want 1.23+

The Fix Lives at the OCI Runtime Layer — Not in Kubernetes

This is the detail that makes "quietly fixed" more than a headline flourish: the fix has no KEP of its own, no feature gate, and ships in nobody's kubernetes release notes, because Kubernetes never owned the buggy code. The conversion from shares to weight happens inside the OCI runtime — runc or crun — at container-creation time, not inside the kubelet or the API server. runc shipped the corrected formula in 1.3.2, released October 3, 2025; crun shipped it in 1.23. The Kubernetes blog post from January 30, 2026 isn't announcing a Kubernetes change — it's documentation, written three months after the actual fix, pointing operators at a runtime-layer change most of them had no reason to have noticed.

That means bumping your Kubernetes minor version does nothing here. What determines whether you're on the fixed formula is which runc/crun binary your container runtime — containerd or CRI-O — is shipping, which in turn is pinned by your node image. Two clusters running the identical Kubernetes version can have completely different cpu.weight behavior for the identical pod spec, purely as a function of when each one last rebuilt its node image.

If you're on a managed offering — GKE, EKS, AKS — you don't control that pin directly; the honest audit is to run the check above against a node from your node pool and see what comes back, then track your provider's changelog for when their default node image picks up runc 1.3.2+ (as of this writing, roll-out timing varies by provider and node-pool image type, so don't assume a recent Kubernetes-version bump pulled the runtime fix in with it). If you operate your own node images — which is the actual audience for the rest of this post — the fix, and the rollout risk, are entirely yours to manage.

Rolling This Out on a Fleet You Own

For a Cluster-API-provisioned fleet — CAPH machines on Hetzner, in bex's case, or any self-managed node pool — "upgrade the runtime" is a node-image change, not a kubectl command, and it has one rollout hazard worth planning around before you touch anything:

  1. Audit first. Run the runc --version / crun --version check from above across every distinct node image currently in the fleet, not just the newest one. A fleet that's been adding capacity over 18 months usually has more image versions in production than anyone remembers provisioning.
  2. Bump the node image, not the running nodes. The formula lives in the runtime binary baked into the image; there's no live patch. Cut a new image pinning runc 1.3.2+/crun 1.23+, and roll it out the way you'd roll out any other base-image change — via MachineDeployment rolling replacement, not an in-place package upgrade on a node you intend to keep trusting.
  3. Expect a mid-rollout priority skew, and canary for it. During a rolling node replacement, the fleet briefly runs both formulas at once. The identical pod spec — same 1000m request — gets cpu.weight = 39 on an old-image node and cpu.weight = 102 on a new one. For most workloads that's a pure improvement and nothing to worry about. But if you're running anything that was implicitly tuned around the old, weaker priority — a background job deliberately given a small request because it "only needed a little CPU and the priority didn't seem to matter much" — that job's actual claim on the CPU roughly triples the moment it lands on a fixed-formula node next to Guaranteed-tier pods that also just got their priority corrected. Canary the new image on a node pool carrying mixed QoS classes before fleet-wide rollout, and watch for anything that was quietly relying on being deprioritized.
  4. Verify with the same command you audited with. After rollout, re-run the cpu.weight check against a live pod on a replaced node and confirm it matches the "fixed" column of the table, not the "buggy" one. Don't verify by reading the node image's changelog — verify the same way the original bug was found, by reading the file the kernel is actually enforcing.

Worth a one-line disambiguation since fleet upgrade planning tends to lump every cgroup-related Kubernetes change into one bucket: this CPU-weight fix is unrelated to the separate, actually-breaking change landing in Kubernetes v1.37, where the kubelet refuses to start at all on a cgroup v1 host unless failCgroupV1: false is explicitly set. That one is a hard boot-time gate you'll notice immediately. This one is a silent scheduling-priority number that never throws an error, never fails a health check, and will sit wrong indefinitely unless you go looking for it — which is exactly why it's worth a dedicated audit pass rather than assuming it rides along with some other upgrade you were already planning.

The Broader Lesson: Requests Are a Contract With Two Enforcers

The uncomfortable generalization here isn't really about cgroups. It's that "the scheduler admitted my pod's request" and "the kernel is enforcing the priority that request implies" are two different guarantees, made by two different layers, and only one of them is visible through the Kubernetes API you're used to querying. Admission and bin-packing happen against the numbers in the pod spec. Actual CPU time-slicing happens against a completely separate number, computed once by a runtime binary you didn't write and rarely think about, then never revisited.

For a platform team running its own node images — rather than treating "Kubernetes version" as the only variable that matters — that's the practical takeaway: the container runtime pinned into your base image is not an implementation detail you can leave un-audited. It's the thing standing between the CPU priority you configured and the CPU priority a workload actually gets.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on Cluster-API-provisioned machines you own, with node images bex tracks and rolls out for you instead of leaving runtime-layer bugs like this one for a tenant to discover by hand. Star the repo on GitHub or deploy your first app today.

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex