Dora NodaOn September 7, 2026, the CNCF named China Merchants Bank — one of China's largest commercial banks — the winner of its End User Case Study Contest at the Shanghai joint summit, for doing something most platform teams still treat as two separate jobs: running AI training and AI inference on one Kubernetes stack. Nearly 10,000 heterogeneous accelerator cards, one control plane built from Kueue, KEDA, Prometheus, HAMi, and Fluid, and a scoreboard that is hard to argue with: average accelerator utilization up from 35% to more than 60%, inference cost per million tokens down more than 60%, and fine-tuning density up fivefold.
Here is the verdict up front, for anyone running GPU-backed workloads on clusters you own: the single-stack bet pays when you share the infrastructure but decouple the runtimes — queue-gated training admission on one path, demand-driven elastic inference on the other, fractional accelerator allocation underneath both. That is the whole post in one paragraph — the rest is the receipts, the fairness rules that keep the two paths from eating each other, and what transfers to a fleet far smaller than a bank's:
| Claim | Number | Scope and baseline |
|---|---|---|
| Accelerator utilization | 35% to more than 60% | Across the ~10,000 cards brought under the unified framework (99% of the bank's accelerator fleet) |
| Inference cost per 1M tokens, input plus output combined | Down more than 60% | Under comparable model and service conditions |
| Multi-tenant fine-tuning density | 5x, with 80% less accelerator usage | Scoped to the Twinkle default: five LoRA tenants sharing one base-model instance |
| Prior trajectory (July 2026 case study) | 100% hardware pool utilization; 30% less cross-machine scheduling | Topology-aware scheduling on the earlier HAMi-based platform this win extends |
Two things to read off this table before the detail. First, every number ships with its scope — the 5x density figure belongs to one specific LoRA-sharing setup, not to "all training everywhere," and the cost cut is measured under comparable conditions, not across different model generations. Second, this is a second data point on a trajectory, not a debut: the bank's earlier HAMi case study already showed full pool utilization. Treat the numbers as the output of a sustained, multi-stage program, not a weekend migration.
Why training and inference fight over the same hardware
Training and inference place significantly different demands on identical silicon, and that mismatch is the entire reason teams split them apart. Distributed training wants stable, predictable capacity — and then sits idle while gang-scheduled workers wait for the remaining cards to free up. Online inference wants the opposite: fast elastic scale against unpredictable traffic, with expensive cards burning money in every idle minute between spikes.
Separate clusters resolve the fight by giving each side its own room, but they strand capacity by construction. Training's reserved-but-waiting cards can't serve inference spikes next door; inference's over-provisioned headroom can't backfill a queued training job. At roughly 10,000 cards, that stranded capacity is a budget line item, not a rounding error — which is why the bank's starting utilization of 35% should look familiar to anyone who has ever sized a GPU pool for peak inference while training queues starved beside it.
The single-stack alternative only works if the sharing has rules. Uncontrolled colocation of batch training and latency-sensitive serving is how you get both sides missing their SLOs simultaneously. So the interesting part of this case study isn't that one scheduler serves both — it's the fairness machinery that makes sharing safe.
The composable stack, component by component
There is no single product here. The bank's AI infrastructure team composed five CNCF projects plus one in-house framework, each owning a distinct job:
| Component | Job in the unified stack |
|---|---|
| Kueue | Training admission, queues, and quotas — jobs don't reserve capacity before they can actually use it |
| KEDA plus Prometheus | Online inference autoscaling driven by live demand signals |
| HAMi | Fine-grained shared accelerator allocation across both paths |
| Fluid | Accelerated access to datasets, model weights, and checkpoints, so cards spend less time waiting on data |
| Twinkle (in-house) | Multi-tenant fine-tuning: five LoRA tenants share one base-model instance by default |
The governing principle, in the bank's own words from AI Infrastructure Architect PeiXiang Tan, is that "training and inference follow workload-specific paths while working in close coordination" on a shared foundation of pooled heterogeneous compute, scheduled training, elastic inference, accelerated data access, and end-to-end observability. Shared infrastructure, decoupled runtimes — the sentence from the hook, now with names attached.
HAMi deserves a gloss because it is the least famous name on the list and the load-bearing one. Formerly known as k8s-vGPU-scheduler and now a CNCF Incubating project, HAMi lets platform teams allocate fractions of a physical accelerator by memory, core count, or device count, enforce per-workload memory and compute limits where the device backend supports it, and schedule pods with topology-aware, binpack, or spread policies across heterogeneous hardware — NVIDIA GPUs alongside NPUs, DCUs, MLUs, and other domestic accelerators. In the bank's stack it is the device-abstraction layer: the thing that turns "10,000 cards from competing vendors" into one schedulable pool both paths can draw from in fine-grained units.
The fairness rules that make sharing safe
This is the section the headline numbers depend on, so here it is stated plainly. Three mechanisms keep batch training from starving latency-sensitive serving and vice versa:
Training can't squat. Kueue manages admission with queues and quotas, so a distributed training job doesn't reserve capacity before it can use it. That single rule kills the most wasteful failure mode of gang scheduling — workers holding cards idle while waiting for the rest of their gang — and it converts training demand from "capacity I grabbed" into "capacity I was granted," which is what makes the remainder safely lendable to inference.
Serving scales from demand, not from reservations. KEDA and Prometheus drive online inference replicas from live traffic signals rather than from pre-provisioned headroom. Inference doesn't own cards; it borrows them when traffic arrives and releases them when it ebbs. The two paths therefore contend only at moments of genuine simultaneous demand, not structurally all day.
The device layer is fractional and workload-aware. HAMi allocates shared accelerator capacity in fine-grained units across both paths instead of dealing whole cards to whichever path asked first. Combined with the topology awareness proven in the bank's earlier case study — which cut cross-machine scheduling for distributed training by 30% — placement stays communication-efficient even as the pool fills past the old 35% utilization equilibrium.
One honest boundary: the announcement doesn't publish the preemption specifics — which priority classes fire when genuine simultaneous contention hits, or how a latency-sensitive serving burst reclaims cards from a running training job. Don't invent those details from this post.
What the bank did publish is arguably more informative: its roadmap treats fairness as still under construction, with dynamic multi-tenant training concurrency, unit-cost-based capacity management fusing utilization, queue-state, and latency signals, and KEDA extended toward serverless inference that scales to zero between spikes. Read that list as an admission that today's fairness rules are good enough for 60% utilization but not the final form — exactly the posture you'd want from a team running other people's money on shared silicon. As CNCF CTO Chris Aniszczyk put it, the case shows "composable, vendor neutral infrastructure" delivering "measurable efficiency at scale" — composable being the operative word, since every one of these rules can be tightened independently.
What transfers to a fleet 1,000x smaller
No self-hosted PaaS team is operating 10,000 heterogeneous cards under financial regulation, and nothing in this case study asks you to. The transferable part is the pattern, and it arrives at a convenient moment: AI-agent sandboxes blur the same batch-versus-serving line the bank just erased. An agent fleet runs long-horizon batch-ish work (evals, bulk sandbox provisioning, background fine-tuning) next to latency-sensitive serving (interactive agents, inference endpoints) on the same nodes — the same two demand shapes, at 1/1000th the scale.
Borrow these four things, roughly in order of cheapness:
- Admission before capacity. Even without Kueue, a queue-and-quota gate in front of batch-style agent work — evals don't start until their full resource ask is grantable — frees the remainder for serving without any new infrastructure.
- Elastic serving from demand signals. KEDA plus Prometheus is off-the-shelf CNCF kit; event-driven scale-to-zero for spiky inference endpoints is the cheapest utilization win on this entire list.
- Fractional device sharing. If your roadmap has any shared GPU pool at all, HAMi's allocate-by-memory model (now under open CNCF governance at Incubating maturity) beats whole-card assignment the moment two tenants share one card.
- Data locality as utilization work. Fluid's lesson generalizes: accelerators idle on data fetch are utilization you already paid for. Faster dataset, weight, and checkpoint access pays back on any fleet with a GPU.
Skip the bespoke parts until demand forces them. Twinkle-style shared-base-model fine-tuning is a response to many LoRA tenants, not a starter project; unit-cost capacity management is what you build after utilization stops being embarrassing. And keep the bank's sequencing: pool first (their 100%-utilization case study came first), unify second, optimize unit cost third. The roadmap order is the adoption order.
The direction of travel
The bank's four roadmap items point one way: toward treating compute as a fungible utility with a per-unit price. Dynamic tenant concurrency, capacity managed against unit cost instead of raw utilization, inference that scales to zero, and ever-broader heterogeneous accelerator support under HAMi — each step dissolves another boundary between "training infrastructure" and "inference infrastructure" until only workload-shaped demand remains.
That is the real content of the single-stack bet. Not that one cluster can technically run both — Kubernetes could always do that — but that with admission queues, elastic scaling, fractional devices, and data acceleration composed underneath, the operational distinction between the two stops being worth a second fleet. A regulated bank just proved it at 10,000 cards. The rest of us get to copy the composition at whatever scale we actually run.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
