Volcano's Headlamp Plugin Kills the vcctl Guesswork: A Practical Walkthrough for Gang-Scheduling AI-Agent Batch Jobs on a Cluster API Fleet

A distributed job that only half-schedules doesn't fail loudly. It just sits there, holding GPUs or CPU cores it can never use, while the rest of the cluster waits for capacity that's already gone. One production case study tracking a 64-GPU training cluster over six months found this exact failure mode drove average GPU idle time to 38 percent — nearly two in every five GPU-hours burned on jobs that couldn't actually run. Fixing the admission model (all pods start together or none do) cut that down to 6.2 percent.

That's not a niche problem anymore. Every self-hosted platform that lets AI agents kick off batch work — bulk sandbox provisioning, eval sweeps, fine-tuning runs — is one badly-timed burst away from the same kind of silent GPU waste. This post is a walkthrough of the two things that actually fix it on Kubernetes: Volcano's gang-scheduling and queue-fairness model, and the new official Headlamp plugin that finally makes that model visible without living in vcctl output. We'll close with a direct answer to the question that actually matters for a Cluster API fleet: is this worth running over hand-rolled PriorityClass rules, or is it more machinery than you need?

Why kube-scheduler alone lets this happen​

The default Kubernetes scheduler binds one pod at a time. It has no concept of "this pod is only useful if four other pods also get a node." For a stateless web service, that's irrelevant — pods come up independently and nothing breaks if they trickle in one by one.

For a distributed training job, a multi-worker eval sweep, or a fleet of AI-agent sandbox pods that need to coordinate, it's a real bug waiting to happen. If three of four workers land but the fourth can't find a node, the job doesn't run — it hangs, holding three nodes' worth of resources hostage while contributing zero useful work. Under enough concurrent batch load, that's how you get to 38 percent idle GPU time: not because the cluster lacks capacity, but because the scheduler keeps handing out partial allocations that never complete.

Volcano, a CNCF batch-scheduling system, exists specifically to close that gap, and it's been production-grade for years — running underneath Spark-on-Kubernetes, Kubeflow training jobs, and Ray clusters. It adds two things kube-scheduler doesn't have: gang scheduling (all-or-nothing admission) and hierarchical queue fair-share (multi-tenant quota with borrowing). Here's what both actually look like in a manifest.

Gang scheduling: minAvailable as the atomic unit​

A VolcanoJob (vcjob) wraps one or more tasks, each a pod template, under a single minAvailable. The scheduler will not bind any pod in the job until it can bind enough of them, across all tasks, to hit that number:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: agent-eval-sweep
spec:
  minAvailable: 4          # gang size: all 4 workers or none
  schedulerName: volcano
  queue: tenant-eval-batch
  tasks:
    - name: worker
      replicas: 4
      template:
        spec:
          containers:
            - name: eval-worker
              image: registry.internal/agent-eval:2026.07
              resources:
                requests: { cpu: "2", memory: 8Gi, nvidia.com/gpu: "1" }
                limits:   { cpu: "2", memory: 8Gi, nvidia.com/gpu: "1" }
          restartPolicy: OnFailure

If only 3 of the 4 GPUs are free, Volcano leaves the whole job pending instead of starting 3 workers and stranding them. That's the mechanism behind the idle-time fix above: no partial admission means no partial allocation sitting idle.

Queues: capacity, deserved, and guarantee​

Volcano's Queue is a cluster-scoped CRD, decoupled from namespaces, so multiple tenants' jobs can share one fairness domain. Three fields do the actual work:

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: tenant-eval-batch
spec:
  weight: 1
  capability:              # hard ceiling this queue can ever consume
    cpu: "64"
    nvidia.com/gpu: "16"
  guarantee:                # reserved floor — never reclaimed by other queues
    resources:
      cpu: "8"
      nvidia.com/gpu: "2"
  deserved:                  # share above which excess can be reclaimed under load
    cpu: "32"
    nvidia.com/gpu: "8"

guarantee is the floor a tenant's steady-state work can always count on. deserved is the fair-share line — a tenant can burst above it while the cluster is idle, but Volcano can reclaim the excess back down to deserved when another queue needs its share. capability is the absolute ceiling regardless of how idle the rest of the cluster is. Volcano computes this with dominant resource fairness (DRF), extended to hierarchical queues (HDRF) when queues nest, so a parent queue's deserved always bounds the sum of its children's.

That's the whole model: gang admission stops partial scheduling from wasting capacity, and capability/guarantee/deserved stop one tenant's batch burst from starving another's. Neither exists in stock kube-scheduler.

Before the plugin: diagnosing a stuck gang by hand​

Knowing the model doesn't mean the model is easy to operate. Before the Headlamp plugin, answering "why is agent-eval-sweep stuck" meant stitching together output from at least three separate commands:

1. vcctl vjob view agent-eval-sweep — job-level phase, but not why it's pending
2. vcctl queue view tenant-eval-batch — queue capacity numbers, in a separate terminal, cross-referenced by hand against the job's requests
3. kubectl describe pod <each-of-4-workers> — per-pod scheduling events, repeated once per task replica

None of these show the relationship between the job, its PodGroup (Volcano's internal gang-tracking object), and the queue that's actually gating admission. An operator has to hold that graph in their head.

After the plugin: one page, four views​

The official Volcano plugin for Headlamp (Kubernetes' maintained successor to the deprecated Kubernetes Dashboard) shipped in June 2026 and replaces that CLI archaeology with four linked views:

The practical win is the Map view specifically: it's the first place an operator can see, in one screen, that agent-eval-sweep's PodGroup is stuck at 3-of-4 because tenant-eval-batch has hit its deserved ceiling — a fact that used to require manually matching two separate vcctl outputs against each other.

Applying it to AI-agent batch jobs on a Cluster API fleet​

The scenario this actually targets on a self-hosted PaaS: an AI agent (or a fleet of them) triggers bursty batch work — provisioning a bank of sandbox pods to run an eval suite, kicking off a fine-tune, running a bulk migration check across every tenant's app. That work is exactly gang-shaped (the sandbox pool or eval workers need to come up together) and exactly multi-tenant (agent A's burst shouldn't starve agent B's steady-state deploys).

Map that onto the two primitives above: each agent's batch run becomes a vcjob with minAvailable set to the sandbox/worker count it actually needs concurrently, submitted to a per-tenant Queue with a guarantee that protects the tenant's normal deploy traffic and a deserved ceiling that caps how much of the shared pool one tenant's eval sweep can burst into. Headlamp's Queue and PodGroup views become the answer to "why is tenant B's batch job pending" without needing vcctl at all — read the Queue view's deserved vs. allocated gap, or the PodGroup view's minAvailable progress, directly.

The part that's easy to miss on a Cluster API fleet specifically: gang admission is instant, node provisioning isn't. Volcano's scheduler decides in one scheduling cycle whether enough already-Ready nodes exist to satisfy a PodGroup's minAvailable. If they don't, and Cluster API's autoscaler needs to spin up new Hetzner machines to cover the gap, that's minutes of MachineDeployment provisioning and node join time — the PodGroup sits Pending the entire time, not because Volcano is broken, but because the compute it's waiting on hasn't finished booting. Two consequences follow directly:

• Size a queue's capability at or below what the current node pool (not the autoscaler's theoretical ceiling) can satisfy in one gang-admission pass, or budget for that provisioning window as visible pending time rather than a scheduling failure.
• The PodGroup view's "waiting on minimum resources" state is your signal to check CAPI's MachineDeployment status, not Volcano's queue math — conflating the two is the most common cause of "why won't this job start" confusion on a fleet where node count isn't fixed.

Bex.co's deploy/rollback surface already treats infrastructure state as something an agent reads and acts on directly rather than through a dashboard-only workflow — the same principle applies here: an agent kicking off a batch eval run should be able to check queue headroom before submitting the gang, not discover the Pending state after the fact.

Does Kubernetes 1.36's native gang scheduling make this moot?​

It's a fair question, and worth answering directly rather than ignoring: Kubernetes v1.36 ("Haru," April 2026) shipped alpha Workload-Aware Scheduling — a Workload API plus a PodGroup concept the Job controller can create natively, gated behind WorkloadWithJob, GangScheduling, and WorkloadAwarePreemption feature flags, all off by default. It gives the built-in Job controller gang admission without installing anything extra.

That's real, and worth tracking. It is not, today, a replacement for Volcano on a multi-tenant fleet, for two concrete reasons:

1. It's alpha and opt-in. All three feature gates ship disabled by default in 1.36 — this is not something you flip on in a production fleet yet, and the KEP timeline for graduation to beta/GA isn't set.
2. It has no hierarchical fair-share queue model. WAS gives you gang admission and workload-aware preemption. It has nothing equivalent to Volcano's capability/deserved/guarantee quota hierarchy, DRF/HDRF fairness, or reclaim-on-overuse semantics — and nothing equivalent to a maintained visual UI for any of it.

If all you need is "start these pods together or not at all," native WAS will eventually cover that for free. If you need "and don't let one tenant's burst starve another's floor," that's a queue-fairness problem WAS doesn't touch, and Volcano is the production-ready answer to it today.

The actual verdict: when is this worth running?​

Back to the question the topic actually poses: is Volcano — now with a real visual debugging story — a realistic default over hand-rolled PriorityClass rules for a Cluster-API-managed fleet running AI-agent batch workloads?

Yes, once you have more than one tenant or workload class competing for the same batch capacity. PriorityClass gives you preemption ordering — higher-priority pods bump lower-priority ones — but nothing else: no gang admission (a preempted low-priority job still schedules pod-by-pod), and no quota hierarchy (nothing stops one workload from consuming the entire node pool if it happens to hold priority). Volcano's added operational cost — running its scheduler and CRDs alongside kube-scheduler — buys you both properties at once, and the Headlamp plugin removes what used to be the real adoption tax: debugging a gang-scheduling failure without a UI.

No, if you're running a single tenant with occasional, non-competing batch jobs. A solo fleet with no fairness contention doesn't need hierarchical queues, and plain PriorityClass plus a cron job stays simpler and sufficient. The crossover point is contention, not scale — the moment a second tenant's batch burst can meaningfully delay a first tenant's, queue fairness stops being optional and hand-rolled priority ordering stops being enough.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with infrastructure state an AI agent can read and act on directly rather than through a dashboard-only workflow. Star the repo on GitHub or deploy your first app today.

Sources​

• Inspect Volcano workloads faster with Headlamp — official Kubernetes blog, June 25, 2026
• Volcano project site and VolcanoJob docs
• Volcano Queue Resource Management docs and Hierarchical Queue docs
• Volcano v1.15 release notes — gang-granularity preemption, DRA queue quota
• Kubernetes v1.36: Advancing Workload-Aware Scheduling — official Kubernetes blog
• Kubernetes Kueue for MLOps: Managing Job Queues and Resource Fairness in Shared Clusters — source of the 38%→6.2% GPU idle-time case study (Kueue, not Volcano; cited as evidence of the class of problem gang scheduling solves).