Deploying vLLM as Just Another Git-Push App: What GPU Scheduling Actually Adds

Push a repo, get a running HTTPS service — that's the whole pitch of a git-push PaaS, and it holds up fine for a stateless web app on CPU and memory requests. Push a repo that starts vllm serve meta-llama/Llama-3.1-8B-Instruct instead, and the pitch runs into a GPU.

The short answer: placement mostly reduces to "just another app" — a resource field and a node-pool label get you there. Routing does not. A stateless web app can go to any healthy replica; a model server can't, because each replica is carrying gigabytes of KV cache state that a round-robin proxy has no way to see. That's the one piece a git-push PaaS's existing stack — Cluster API for node provisioning, Kueue-style admission, a CPU/memory-only ingress — doesn't already have an answer for. Here's what changes at each layer, worked against bex's actual Hetzner GPU inventory instead of a hypothetical H100 fleet.

The Manifest Diff​

A normal git-push deployment on bex asks for CPU and memory:

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2"
    memory: "1Gi"

A vLLM deployment adds exactly one new resource type and one new scheduling constraint:

resources:
  requests:
    cpu: "4"
    memory: "16Gi"
    nvidia.com/gpu: "1"
  limits:
    cpu: "8"
    memory: "24Gi"
    nvidia.com/gpu: "1"
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: nvidia.com/gpu.product
              operator: In
              values: ["NVIDIA-RTX-4000-SFF-Ada-Generation"]

nvidia.com/gpu isn't a built-in Kubernetes resource type — it's advertised by the NVIDIA device plugin, which runs as a DaemonSet on every GPU node and reports the card as an allocatable resource to the kubelet. nvidia.com/gpu.product isn't built in either; it's a node label written by GPU Feature Discovery, part of the NVIDIA GPU Operator, and it's what makes node affinity possible in a fleet with more than one GPU model. Both requests and limits are set to the same integer, because vLLM doesn't fractionally consume a GPU the way a container fractionally consumes CPU — it either gets a whole card (or a whole MIG slice, more on that below) or it doesn't run.

That's a real addition to the manifest schema — two new fields — but it's still a manifest field, not a new deployment model. The harder question is which node that affinity rule should actually point at.

Which Node, on Hardware bex Actually Owns​

bex runs on owned Hetzner machines, not an elastic H100 fleet, so "which GPU" isn't an abstract choice — it's a choice between two specific SKUs Hetzner currently sells as dedicated GPU servers:

Neither is an H100 or an L40S — Hetzner doesn't sell those. That constraint turns out to set the actual sizing question: which model fits which box.

At fp16, a model needs roughly 2 GB of VRAM per billion parameters, plus 15–20% overhead for KV cache and activations. A 7B model at fp16 lands around 14–16 GB — it fits on a GEX44's 20 GB card with headroom to spare, and needs no quantization to do it. A 70B-class model at fp16 needs on the order of 140 GB, which doesn't fit on either box; even the GEX131's 96 GB card is short. Getting a 70B model onto a single Hetzner GPU server at all means 4-bit quantization (GPTQ/AWQ), which brings the footprint down to roughly 42–45 GB — comfortably inside a GEX131, and nowhere close to fitting a GEX44 regardless of quantization.

So the node affinity rule from the manifest above isn't decorative — nvidia.com/gpu.product: NVIDIA-RTX-4000-SFF-Ada-Generation versus NVIDIA-RTX-PRO-6000-Blackwell-Max-Q-Workstation-Edition is the difference between "this pod schedules" and "this pod sits Pending forever because no node in the fleet has enough VRAM." A tenant pushing a 7B model and a tenant pushing a quantized 70B model need to land on genuinely different node pools, and the platform has to know that before the pod is ever created — not discover it from an OOM-killed container.

MIG Isn't a Fleet-Wide Feature — It's a GEX131-Only Feature​

The other reason those two SKUs aren't interchangeable: only one of them supports hardware GPU partitioning.

NVIDIA's Multi-Instance GPU (MIG) splits a single card into up to seven isolated instances, each with its own memory, cache, and compute allocation — a crash or memory overflow in one instance can't touch another. The RTX PRO 6000 Blackwell in the GEX131 supports MIG, up to four instances, managed declaratively through the GPU Operator's MIG Manager component. The RTX 4000 SFF Ada in the GEX44 does not — Ada-generation workstation cards were never MIG-capable, full stop.

Sharing a GEX44 across tenants means falling back to time-slicing: the GPU Operator can still let multiple pods take turns on the same card, but with no memory isolation and no fault isolation. A workload that overruns its VRAM budget or hangs on a GEX44 can take a neighboring tenant's inference requests down with it. On a GEX131, the same failure stays inside its MIG slice.

That's not a tuning knob a platform can average away — it's a hard capability boundary that has to be visible in the node-pool primitive itself, not buried in a shared "GPU node" label. A tenant asking for isolation-guaranteed multi-tenancy needs to land on a GEX131 MIG slice; a tenant fine with best-effort sharing on cheaper hardware can go on a GEX44 under time-slicing. Collapsing both into one has-gpu: true label would mean silently promising an SLA the hardware can't back on half the fleet.

From Bare Metal to a Schedulable Resource: What CAPH Actually Adds​

bex provisions nodes through Cluster API Provider Hetzner (CAPH), which manages Hetzner's dedicated Robot servers — the GEX44/GEX131 line — as HetznerBareMetalHost inventory: a server gets registered, then attached to a HetznerBareMetalMachineTemplate inside its own MachineDeployment. This is the same mechanism bex already uses for every CPU/memory-only node pool; a GPU node pool is a second MachineDeployment pointed at a different bare-metal template, not a different provisioning path. CAPH's kubeadm bootstrap provider generates the same cloud-init join sequence for a GEX131 that it does for an ordinary worker.

What's genuinely new happens after the node joins, and it isn't part of CAPI's job at all: the NVIDIA GPU Operator — deployed once, cluster-wide, as its own Helm release — detects the new node's GPU via Node Feature Discovery, installs the driver and container toolkit, starts the device plugin and MIG Manager, and only then writes the nvidia.com/gpu.product label the manifest above depends on. Until that label lands, the node is CAPI-managed and kubelet-ready but functionally invisible to any GPU-aware scheduling decision.

Kueue is the layer that turns that label into an admission boundary. A ResourceFlavor keyed to it gives each GEX SKU its own quota pool:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: gex131-blackwell
spec:
  nodeLabels:
    nvidia.com/gpu.product: NVIDIA-RTX-PRO-6000-Blackwell-Max-Q-Workstation-Edition
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: gex44-ada
spec:
  nodeLabels:
    nvidia.com/gpu.product: NVIDIA-RTX-4000-SFF-Ada-Generation

That's the concrete shape of "GPU-aware node-pool affinity": one more MachineDeployment per GPU SKU (a CAPI-native change bex's provisioning layer already knows how to make), one cluster-wide GPU Operator install (a one-time cluster addition, not a per-tenant one), and one Kueue ResourceFlavor per SKU tying node labels to a quota pool that CPU/memory bin-packing never needed. None of it requires a new deployment model — it composes with the MachineDeployment-per-pool pattern the fleet already runs.

The Failure Mode That Only Shows Up Under Real Traffic​

Every piece so far is a placement problem, solvable at pod-creation time. The failure mode that isn't shows up only once multiple tenants are actually sending inference traffic through a shared pool: naive load balancing sends each request to any healthy replica, and multiple industry reports on shared inference fleets converge on the same number for what that costs — utilization stuck around 40% even on teams running mature stacks, against 90%+ once requests are routed with actual GPU state in mind. The gap isn't idle hardware sitting unused; it's live GPUs sitting mid-request-queue while a round-robin proxy sends the next request to a replica that's already backed up, because the proxy has no visibility into per-replica load.

For vLLM specifically, the sharper version of that problem is KV cache. A model server keeps recently-processed prompt prefixes cached in GPU memory so a follow-up request sharing that prefix doesn't have to recompute it. Route two requests that share a long prefix to two different replicas, and both pay the full compute cost — a cache miss neither one needed to take. This is where a git-push PaaS's existing ingress genuinely runs out of road: a stateless proxy that treats every backend as interchangeable is the right design for a CPU/memory web app and the wrong one for a model server with per-replica state. The fix in the current stack is the Gateway API Inference Extension's endpoint picker (EPP), which tracks each replica's KV cache utilization and queue depth and routes new requests to whichever replica already holds the matching prefix — one production deployment cited a 3x improvement in output tokens/second and a 2x reduction in time-to-first-token after turning prefix-aware routing on.

That's the honest answer to whether "just another git-push app" holds up: for scheduling and placement, yes — a resource field, a node-affinity rule, and a Kueue ResourceFlavor per GPU SKU get a vLLM deployment scheduled correctly on hardware a platform already owns. For routing, no — a model server needs a KV-cache-aware gateway extension that a stateless web app never asked for, and skipping it doesn't just cost a few percent of throughput; it's the difference between the utilization numbers above.

What This Means for a Tenant Pushing vLLM Today​

Concretely, closing that gap for a bex tenant means three additions to the existing git-push flow, in order of how much they change the platform:

• A nvidia.com/gpu resource field in the app manifest — small. It's one more resource type Kueue and the scheduler already understand how to admit.
• A GPU-labeled node-pool affinity rule resolved against the fleet's actual SKUs — GEX44 versus GEX131, with MIG-capable pools kept distinct from time-sliced ones. Medium: a provisioning and labeling concern, not a new deployment model, and CAPH's MachineDeployment-per-pool pattern already covers it.
• An inference-aware routing layer in front of any multi-replica model server — the actual new piece. Nothing in a stateless-app ingress fills this gap on its own.

None of it requires bex to become a managed inference platform or abstract away which physical GPU a tenant's pod lands on — the fleet stays Hetzner-owned bare metal, and a tenant pushing a vLLM-serving repo still gets there through the same git push, node pool, and quota primitives every other app on the platform uses. It's the same shape, with a hardware-aware placement rule and a smarter proxy layered on top.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• Resource Flavor — Kueue.
• GEX44 — Hetzner.
• GEX131 — Hetzner.
• MIG Support in Kubernetes — NVIDIA.
• RTX PRO 6000 Blackwell Workstation Edition — NVIDIA.
• Gateway API Inference Extension — kubernetes-sigs.
• Prefix Cache Aware Plugin — Gateway API Inference Extension.
• cluster-api-provider-hetzner — syself.
• Using Kubernetes — vLLM Documentation.