Kueue 1.3 Plus JobSet: Skip the Second Scheduler for Agent Batch Jobs — Until You Can't

Every team that runs AI-agent sandboxes on its own Kubernetes fleet eventually hits the batch-scheduling question. One tenant's overnight eval sweep lands at the same time as another tenant's bulk sandbox provisioning, the cluster has finite GPUs and CPUs, and somebody has to decide which jobs wait, which jobs preempt, and which jobs run as an all-or-nothing gang. The traditional answer is Volcano: install a second scheduler with its own queues, PodGroups, and fair-share plugins, and operate it forever next to kube-scheduler. In April 2026, Kueue 1.3 made the alternative credible by adding built-in support for JobSet workloads alongside new v1beta2 APIs, completing a native two-piece batch stack — Kueue for quota and admission, JobSet for gang coordination — that runs entirely on top of the default scheduler you already have.

Here is the verdict up front, with the evidence behind it in the sections below:

The rest of this post shows exactly where that line sits, with the manifests to prove it.

How the two pieces divide the work​

Kueue and JobSet solve different halves of the batch problem, which is why they compose so cleanly. Kueue never schedules a single pod. It acts as a gate: a workload (a Job, a JobSet, a RayJob, a plain Pod group) is created suspended, Kueue admits it against quota when capacity and priority allow by flipping spec.suspend to false, and the default scheduler then places the pods normally.

The quota model has three objects worth learning. A ResourceFlavor describes a pool of like hardware (for example, on-demand CPUs versus spot). A ClusterQueue binds flavors to nominal quotas and joins a cohort, and cohorts are where the interesting economics happen: queues can borrow unused quota from their cohort-mates, so a tenant's eval sweep can burst into idle capacity and give it back automatically. A LocalQueue is the namespaced handle tenants actually submit to.

Fair sharing, hierarchical cohorts, preemption, and Prometheus metrics for queue depth and admission latency all ride on top of this model. Kueue's June 2026 v0.18 release added workload-creation latency, eviction latency, and pending-resource metrics for exactly this operational surface.

JobSet, owned by sig-scheduling, is the other half: a controller for managing coordinated groups of Jobs with shared failure and success handling. Think of one agent-eval sweep as ten Jobs that must start together, fail together, and report as one unit — that is a JobSet. Where Kueue answers "is this workload allowed to run now," JobSet answers "how do its pieces run as a gang." A minimal pair looks like this:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: cpu-pool

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: sandboxes
spec:
  cohort: tenants
  nominalQuota:
    - name: cpu
      nominalValue: "40"
    - name: memory
      nominalValue: 160Gi
  resourceGroups:
    - coveredResources: [cpu, memory]
      flavors:
        - name: cpu-pool
          resources:
            - name: cpu
              nominalQuota: "40"
            - name: memory
              nominalQuota: 160Gi

apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: team-evals
  namespace: tenant-a
spec:
  clusterQueue: sandboxes

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: nightly-eval-sweep
  namespace: tenant-a
spec:
  failurePolicy:
    maxRestarts: 2
  replicatedJobs:
    - name: eval-worker
      replicas: 10
      template:
        spec:
          parallelism: 1
          completions: 1
          template:
            spec:
              containers:
                - name: eval
                  image: registry.example.com/eval-runner:2026.09
              restartPolicy: OnFailure

Submit the JobSet to the team-evals queue and the flow is: Kueue holds it suspended until the sandboxes ClusterQueue (borrowing from the tenants cohort if teammates are idle) admits it, then JobSet coordinates the ten workers with shared restart and failure semantics. No second scheduler binary, no CRDs from outside the Kubernetes-native family — Kueue is a CNCF Incubating project and JobSet lives under the kubernetes-sigs umbrella, which matters when you are deciding what your small ops team has to keep upgraded.

What Kueue 1.3 actually changed​

Before 1.3, running coordinated gangs through Kueue meant wiring the integration yourself or accepting that Kueue saw your workload as an opaque Job. The 1.3 release, documented in Red Hat's April 2026 release notes, added first-class support for JobSet workloads (plus LeaderWorkerSet for the inference-pool pattern), so Kueue understands the gang as a unit for admission, borrowing, and preemption decisions rather than admitting its Jobs piecemeal. The same release introduced v1beta2 APIs — the stability signal that the ClusterQueue, LocalQueue, and ResourceFlavor contracts your manifests depend on are settling rather than churning under you.

That combination is the whole thesis of the native stack: admission decisions made against the real unit of work (the whole sweep, not ten loose Jobs), expressed in versioned APIs, enforced without replacing kube-scheduler. The upstream direction confirms the bet — sig-scheduling's Workload-Aware Scheduling work is formalizing the JobSet-as-scheduling-unit pattern — but you do not need any of that future to use 1.3 today.

Head-to-head: Kueue plus JobSet versus Volcano​

Both paths can run your agent batch fleet. The differences are architectural, and they show up in your on-call rotation, not just your YAML:

Read the table against the reference workload — a self-hosted platform provisioning AI-agent sandboxes and eval sweeps for a handful of tenants. Nothing in that workload needs a replacement scheduler: gangs are Job-sized, quota with borrowing covers the noisy-neighbor problem, and the failure mode you actually hit is "whose sweep waits," which is an admission decision, not a placement algorithm.

Volcano earns its keep when partial placement is a correctness problem (a 32-worker DDP run with 31 workers placed is wasted money, not degraded service) or when tenant count grows past what Kueue's cohort admission model was designed to arbitrate. Note the two are not mutually exclusive: several production architectures run Kueue quotas on top of Volcano gangs, taking admission from one and placement from the other. Start native, and if you ever outgrow it, Kueue's quota layer survives the migration.

A decision rule for a Cluster-API fleet​

For a fleet managed declaratively — Cluster API reconciling machines, GitOps reconciling workloads — every extra control-plane component is a manifest you own, an upgrade you schedule, and a 3 a.m. page you accept. That tilts the default hard toward the native stack:

• Fewer than ~20 tenants, batch shaped like sandboxes and sweeps: run Kueue + JobSet. Put each tenant on a LocalQueue, bind the queues to one cohort so idle capacity is borrowable, and set preemption so a latency-sensitive tenant can reclaim its nominal share from a burst neighbor. Your entire batch policy is reviewable YAML in the same repo as everything else.
• Add Volcano when one of three triggers fires: your first multi-node distributed-training workload where gang placement is correctness-critical; tenant growth that needs hierarchical fair-share queueing deeper than cohort borrowing; or a GPU-sharing requirement (fractional allocation, topology-aware packing) that flavor-level quotas cannot express.
• Revisit annually, not quarterly. The native stack is where upstream investment is flowing — JobSet's scheduling integration and Kueue's quota model are both under active sig stewardship — so the boundary moves in Volcano's disfavor over time. The migration path (keep Kueue, swap placement) means you are never locked in by starting simple.

One honest caveat: Kueue's simplicity assumes your tenants' jobs are reasonably well-behaved about requesting what they use. Cohorts with borrowing punish quota-liars gently and quota-honest neighbors harshly if requests are fiction — the same request-hygiene discipline bin-packing already demands applies here, and no scheduler choice fixes lying resource requests.

Conclusion​

Kueue 1.3 plus JobSet is the first Kubernetes-native batch stack that lets a small self-hosted platform say no to a second scheduler with a straight face: quota, borrowing, fair sharing, and gang coordination, all in kubectl-native objects, all on top of the scheduler you already run. Volcano remains the right answer for distributed-training gangs and deep fair-share hierarchies — that is what a decade of batch-scheduler hardening buys — but it is now a deliberate upgrade for a known trigger, not the default tax on running batch at all. If your fleet's batch story is agent sandboxes and eval sweeps, start with the native stack, encode the three triggers above as the documented moment you would switch, and spend the operational budget you saved on something your tenants can see.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• Red Hat Developer, "Red Hat build of Kueue 1.3: Enhanced batch workload management on Kubernetes" (April 16, 2026) — JobSet and LeaderWorkerSet support, v1beta2 APIs. link
• Kueue upstream documentation (kueue.sigs.k8s.io) — ClusterQueue/LocalQueue/ResourceFlavor, cohorts and borrowing, JobSet/Job/JobSet integrations, Prometheus metrics. link
• Michał Żyliński, "Kueue v0.18: What's new?" (June 2026) — workload creation/eviction latency and pending-resource metrics. link
• kubernetes-sigs/jobset KEP: WAS integration (two-phase plan, all-or-nothing gang admission). link
• Kubernetes blog, "Kubernetes v1.37: Advancing Workload-Aware Scheduling" (September 8, 2026) — CompositePodGroup and native JobSet/LeaderWorkerSet scheduling direction. link
• Google Cloud docs, "Implement a job queuing system with quota sharing between namespaces on GKE" — cohorts and borrowing semantics in practice. link
• AWS Labs, global-capacity-orchestrator scheduler comparison — Volcano (batch scheduler, gang) vs Kueue (queue manager, quotas/fair sharing). link
• KodeKloud, "MLOps on Kubernetes: The Complete Guide" — Kueue plus Volcano as complementary layers. link