Your Scheduler Config Has No Staging Environment: Rehearsing Bin-Packing Changes With kube-scheduler-simulator

Every other critical control-plane change in your fleet has a staging environment. Your scheduler configuration does not. A new KubeSchedulerConfiguration — a flipped scoring strategy, a new PodTopologySpread default, a tuned shape curve — goes from a YAML edit straight to production, and the first time you learn what it actually does to pod placement is when real tenant pods start landing on different nodes. For a multi-tenant platform packing many small workloads onto a fixed pool of owned machines, that is a remarkable amount of trust to place in one config file.

There is a fix, and it is an upstream SIG Scheduling project: mirror your fleet's real nodes and pods into kube-scheduler-simulator, apply the config change there, and read the per-plugin placement verdicts before a single production pod moves. The loop is four steps — export, mutate, replay, read — and this post works all four on a representative three-node pool, including the before/after placement table that is the whole point.

Why a dev cluster cannot audition a scheduler change​

The standard advice is to test scheduler changes in a development cluster first. The official simulator announcement says the quiet part out loud: with a limited set of tests, it is impossible to predict every scenario in a real-world cluster, so teams test in a small dev cluster and hope nothing breaks in production. Hope is doing a lot of work in that sentence.

The reason a dev cluster is a poor audition stage is structural, not a matter of effort. Scheduling behavior is a function of contention: which nodes exist, how full each one already is, and which pods are competing for the same headroom at the same time. A two-node dev cluster with three idle test pods exercises none of the scoring math that matters. Bin-packing regressions — the class of failure where a strategy change silently spreads workloads across every node instead of consolidating them, or strands an entire node empty while others fill — only appear once real pods compete for real capacity.

They surface after the fact, in production metrics: node counts that refuse to come down, utilization histograms with a fat tail of half-empty machines, tenants landing next to noisy neighbors they were previously packed away from.

So the operator choosing between scoring strategies is flying blind in exactly the situation where the choice matters most: a fixed fleet of owned machines where every wasted node is a monthly bill that cannot be returned.

The simulator in one paragraph​

kube-scheduler-simulator is a SIG Scheduling subproject that started as a Google Summer of Code 2021 project by Kensei Nakada and was introduced on the Kubernetes blog in April 2025. It runs a fake cluster backed by KWOK (Kubernetes WithOut Kubelet), so nodes behave like nodes for scheduling purposes without any kubelet, runtime, or workload actually running. Inside it, a debuggable scheduler replaces the vanilla scheduler and writes the verdict of every plugin at every scheduling-framework extension point into the pod's own annotations: which nodes passed filtering and why, every plugin's per-node score, the final weighted scores, and the selected node. A web UI visualizes those annotations.

It needs only Docker on a laptop — no cluster at all — and serves on localhost:3000.

Two capabilities make it more than a teaching toy. First, the import feature copies the scheduling-relevant resources — nodes, pods, and the other objects the default plugins actually read — out of a live cluster into the fake one, with continuous syncing so the mirror stays fresh. The stated purpose is direct: simulate deploying a new scheduler version in a production-like environment without impacting live workloads. Second, custom scheduler plugins and extenders can be integrated into the debuggable scheduler, and the debuggable scheduler itself can run standalone on a real cluster or in integration tests. The project's own use-case list names the audience for this post directly: cluster admins assessing how a cluster would behave with changes to the scheduler configuration.

Worked example: mirror the fleet, flip the strategy, read the annotations​

Here is the loop end to end, shaped for a Cluster-API-managed fleet with a fixed Hetzner-style node pool. Imagine three 8-CPU, 32 GB workers at different fill levels — the ordinary, unglamorous state of a bin-packed multi-tenant fleet mid-week:

Step 1 — export. Point the simulator's import at the management or workload cluster and pull nodes plus the running pods that constitute current allocation. What the scheduler's default plugins consider during placement is what gets mirrored, so the fake pool carries the same contention shape as the real one: node-a nearly full, node-c nearly empty.

Step 2 — mutate. Apply the candidate scheduler configuration to the simulator only. The classic candidate is the NodeResourcesFit scoring strategy. Kubernetes ships three: LeastAllocated (the default, which spreads pods toward emptier nodes), MostAllocated (which packs pods onto the fullest node that still fits, the bin-packing strategy documented on the official Resource Bin Packing page), and RequestedToCapacityRatio (a configurable shape curve over the request-to-capacity ratio for operators who want packing with a custom profile).

A candidate config looks like this:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated

Step 3 — replay. Submit the next batch of tenant pods — a deploy wave, the nightly cron surge, the shape of traffic you actually worry about — to the simulator and let the debuggable scheduler place them against the mirrored pool.

Step 4 — read. Open a placed pod and read what the scheduler was thinking. Every extension point gets an annotation, and the two that matter most are the filter verdicts per node and the final scores per node:

metadata:
  annotations:
    kube-scheduler-simulator.sigs.k8s.io/filter-result: >-
      {"node-a": {"NodeResourcesFit": "passed", ...},
       "node-b": {"NodeResourcesFit": "passed", ...},
       "node-c": {"NodeResourcesFit": "passed", ...}}
    kube-scheduler-simulator.sigs.k8s.io/finalscore-result: >-
      {"node-a": {"NodeResourcesFit": "91", ...},
       "node-b": {"NodeResourcesFit": "52", ...},
       "node-c": {"NodeResourcesFit": "18", ...}}
    kube-scheduler-simulator.sigs.k8s.io/selected-node: node-a

Under MostAllocated, the nearly-full node-a outscores the empty node-c by a wide margin and the pod packs in — the simulator shows you the score gap per plugin, so when a placement surprises you, the annotation says exactly which plugin outvoted the others. Run the same replay with the strategy flipped back to the LeastAllocated default and the ranking inverts: node-c wins, the wave spreads, and you have quantified the consolidation difference between the two strategies on your pool, with your pods, before touching production.

That before/after table — same pods, same nodes, two strategies, placements diffed — is the artifact worth building the habit around. Attach it to the scheduler-config PR the way you would attach a benchmark to a performance PR.

The two knobs that decide your packing (and the one that fights it)​

Two scoring inputs decide the consolidation behavior of a fleet, and the replay loop is how you set them deliberately instead of inheriting them.

The first knob is the NodeResourcesFit strategy itself. MostAllocated packs tightest with no tuning, which is what a fixed fleet paying per machine usually wants. RequestedToCapacityRatio lets you draw the packing curve yourself — for example, scoring nodes steeply once they pass 70 percent allocated so the scheduler strongly prefers filling warm nodes but still avoids the pathological very-last-slot placements. The replay catches the difference between "packing in theory" and "packing against my tenants' actual request sizes," which is where a hand-drawn curve either earns its keep or gets simplified back to MostAllocated.

The second knob is NodeResourcesBalancedAllocation, which favors nodes that would end up with balanced CPU and memory utilization after placement. It is the counterweight to naive packing: a pure MostAllocated setup can stack CPU-heavy pods onto one node until its memory sits idle and wasted while other nodes hold the inverse imbalance. The annotations show both plugins' scores side by side, so the replay reveals whether your packing strategy is actually stranding the resource you are not packing on.

And the force fighting both knobs is spread. PodTopologySpread, pod anti-affinity, and topology constraints exist to keep tenants' replicas apart — and every spread rule fragments packing by construction. This is the central tension of multi-tenant scheduling: resilience wants pods apart, economics wants them together. A simulator replay with the fleet's real spread constraints loaded is the cheapest place to see the equilibrium price of a new spread rule — how many extra nodes the same workload needs once the constraint applies — before the constraint ships. If your platform recently touched its spread defaults, re-running the replay is the cheapest way to learn what that change cost in nodes.

What the replay cannot tell you​

An honest runbook names the boundary. The simulator replays scheduling, not execution: it places pods by their declared requests against mirrored allocation, so anything the scheduler cannot see is invisible to the replay. Real CPU/memory utilization versus declared requests, kubelet eviction under memory pressure, GPU device-plugin and Dynamic Resource Allocation behavior, storage and network contention — none of that participates. A placement that looks perfect in the simulator can still degrade on a node whose tenants burst past their requests.

That burst-past-requests failure mode is exactly what pressure-based signals exist to catch: Kubernetes 1.36's GA graduation of PSI metrics and the workload-aware scheduling work maturing through 1.36 into 1.37 both move placement-relevant signals from static requests toward observed behavior, and a request-based replay will not preview those dynamics.

Treat the simulator as a gate for configuration logic — "does this strategy/constraint produce the placements we intend on our pool" — not as a performance model. The placement diff is a necessary check, not a sufficient one.

Make it a gate, not a demo​

The teams that get value from this tool do not open the web UI once. They wire the replay into the change process for scheduler configuration: export the current node and pod set, apply the candidate config in the simulator, replay a standard batch shaped like a real deploy wave, and diff the placements against the current config's replay. A strategy flip that consolidates a wave onto two nodes instead of three is a merged PR with evidence. A new spread default that costs half a node of headroom across the pool is a deliberate, priced decision instead of a surprise in next month's utilization review.

For a self-hosted PaaS on fixed, owned machines, this loop closes the one testing gap the platform cannot buy its way out of. Managed Kubernetes sells you the scheduler as somebody else's tested default; a Cluster-API fleet owns the scheduler configuration the way it owns the machines, and ownership without a rehearsal stage is just optimism. The simulator is the rehearsal stage — upstream-maintained, free, and runnable from a laptop with nothing but Docker.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• Kubernetes blog, "Introducing kube-scheduler-simulator" (April 7, 2025) — kubernetes.io: GSoC 2021 origins, debuggable scheduler, per-extension-point annotations, Docker-only setup, import/sync purpose, the three use cases.
• kubernetes-sigs/kube-scheduler-simulator repository — github.com: KWOK-backed fake cluster, localhost:3000 web UI.
• Simulator docs, "import-cluster-resources.md" — github.com: which resources are imported, continuous production-to-simulator syncing.
• Kubernetes docs, "Resource Bin Packing" — kubernetes.io: MostAllocated and RequestedToCapacityRatio scoring strategies.
• Kubernetes docs, "Scheduler Configuration" reference — kubernetes.io: NodeResourcesFit strategies (LeastAllocated default, MostAllocated, RequestedToCapacityRatio), NodeResourcesBalancedAllocation.