GPU Autoscaling on Kubernetes With KEDA: Scaling on the One Metric HPA Can't See

Picture the alert you will never get: a vLLM pod serving 200 concurrent requests, its GPU completely saturated, request latency climbing — and the Horizontal Pod Autoscaler reporting 8% CPU utilization and doing nothing. That is not a misconfiguration. HPA watches CPU and memory because those are the resources Kubernetes meters natively.

A GPU arrives in the scheduler as nvidia.com/gpu: 1, an integer device count with no utilization percentage attached, so the busiest GPU in your fleet is invisible to the only autoscaler most teams ever configure. The fix is to scale on "is the GPU actually busy" instead, and the practical way to do that today is KEDA — first through the standard metrics pipeline, then, when that pipeline's latency and moving parts start to hurt, through a purpose-built external scaler.

Why HPA is blind to the one resource that matters​

Three facts combine into the blind spot. First, the device-plugin model exposes GPUs as whole, indivisible units: a pod requests nvidia.com/gpu: 1 the way it requests a parking space, and nothing in that request says how hard the device is working. Second, HPA's built-in path only understands CPU and memory via the metrics server; anything else needs a custom-metrics pipeline that you build, secure, and keep highly available yourself.

Third, CPU is a lying proxy for GPU work. Inference spends its time inside CUDA kernels, not the process scheduler, so a pod can sit at single-digit CPU while its GPU is pegged — it never crosses a scale-up threshold during a burst, and it never drops below a scale-down threshold when idle. Scaling GPU workloads on CPU is like scaling a restaurant on parking-lot occupancy while ignoring the dining room.

The signal you actually want already exists: NVIDIA's DCGM exporter publishes per-GPU utilization (DCGM_FI_DEV_GPU_UTIL), memory, temperature, and power as Prometheus metrics from a DaemonSet on every GPU node.

The obvious objection is that HPA already supports custom metrics through the Prometheus adapter, so why add KEDA at all. Three reasons survive contact with production. KEDA scales to zero on an external signal, which HPA's resource metrics fundamentally cannot trigger. KEDA removes the adapter — a stateful, must-not-fail component in front of every scaling decision — in favor of a trigger that queries Prometheus directly.

And KEDA's multi-trigger OR semantics (take the max across utilization and queue depth) express "scale on whichever signal fires first" without chaining HPAs that fight over one replica count. Two HPAs on one deployment is a conflict waiting for a quiet Friday; two triggers in one ScaledObject is the documented happy path.

What follows is the shortest path from that signal to scaling decisions, then the leaner path, then the bill.

The working chain: DCGM to Prometheus to a ScaledObject​

The standard pattern has four hops: dcgm-exporter (DaemonSet) exposes GPU telemetry, Prometheus scrapes it, KEDA's built-in Prometheus scaler queries it, and a ScaledObject turns the answer into replicas — including zero. Here is the whole scaling decision for a vLLM deployment:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-gpu-scaler
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm
  minReplicaCount: 0     # scale to zero when nobody is prompting
  maxReplicaCount: 8     # one GPU per replica caps the blast radius
  pollingInterval: 15    # ask Prometheus every 15s
  cooldownPeriod: 120    # wait 2 min after load drops before scaling down
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        metricName: gpu-utilization
        query: avg(DCGM_FI_DEV_GPU_UTIL{namespace="inference"})
        threshold: "65"
        activationThreshold: "5"

Line by line: minReplicaCount: 0 is KEDA's killer feature here — vanilla HPA still cannot scale on CPU or memory from zero, because those metrics need a running pod to exist. threshold: "65" means one replica per 65 points of average GPU utilization, so a sustained 100% busy GPU grows the deployment. activationThreshold: "5" keeps the deployment at zero until utilization actually lifts off the floor, which stops flapping on exporter noise. And cooldownPeriod is the anti-thrash guard: GPU scale-down that races model load times will oscillate, and every oscillation on a GPU node costs minutes, not seconds.

Two refinements before this is production-shaped. First, scope the query's labels to the workload — which labels the exporter attaches (pod, container, namespace) depends on the dcgm-exporter version, so verify with the query in Prometheus before trusting the average.

Second, add a queue-depth trigger alongside utilization. vLLM exposes vllm:num_requests_waiting, and queue depth sees a burst before utilization does: waiting requests spike the instant arrivals exceed service rate, while utilization is already at 100% and has nowhere to go. KEDA evaluates every trigger and takes the maximum proposed replica count, so the two signals complement rather than fight:

    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        metricName: vllm-queue-depth
        query: sum(vllm:num_requests_waiting{namespace="inference"})
        threshold: "8"             # one replica per 8 waiting requests
        activationThreshold: "1"   # any queue at all wakes the deployment

This chain works, and it is what most teams should build first. Its honest cost is five-ish components and 15 to 30 seconds of metric latency from scrape interval to scaling decision. When that latency or that component count starts to hurt, the next step is not a bigger Prometheus — it is removing Prometheus from the scaling path entirely.

Building the external scaler: utilization plus queue depth, no Prometheus​

KEDA's escape hatch is the external scaler: any gRPC server implementing the externalscaler.ExternalScaler service (IsActive, GetMetricSpec, GetMetrics, plus streaming StreamIsActive) can drive scaling decisions, and any keys you put in the trigger's metadata are forwarded to it verbatim. That means a ~70-line Python server can answer KEDA directly from live telemetry. The sketch below assumes the common self-hosted starting point — one GPU node, so per-node NVML reads are the fleet's reads — and serves both signals from the previous section: local GPU utilization via NVML, and global queue depth scraped straight from vLLM's own /metrics endpoint, no Prometheus server involved.

# gpu_external_scaler.py — minimal KEDA external scaler.
# Dependencies: grpcio, pynvml, requests. Proto stubs generated from
# kedacore/keda's externalscaler.proto (IsActive, GetMetricSpec, GetMetrics).
import grpc
from concurrent import futures
import pynvml, requests
import externalscaler_pb2 as pb
import externalscaler_pb2_grpc as rpc
 
VLLM_METRICS = "http://vllm.inference.svc:8000/metrics"
UTIL_THRESHOLD, QUEUE_THRESHOLD = 70.0, 8.0
 
def max_gpu_util() -> float:
    pynvml.nvmlInit()
    return max(
        pynvml.nvmlDeviceGetUtilizationRates(
            pynvml.nvmlDeviceGetHandleByIndex(i)).gpu
        for i in range(pynvml.nvmlDeviceGetCount()))
 
def queue_depth() -> float:
    for line in requests.get(VLLM_METRICS, timeout=5).text.splitlines():
        if line.startswith("vllm:num_requests_waiting "):
            return float(line.split()[1])
    return 0.0
 
class GPUScaler(rpc.ExternalScalerServicer):
    def IsActive(self, req, ctx):
        # Scale-from-zero: active if EITHER signal shows work.
        return pb.IsActiveResponse(
            result=max_gpu_util() > 5 or queue_depth() > 0)
 
    def GetMetricSpec(self, req, ctx):
        return pb.GetMetricSpecResponse(metricSpecs=[
            pb.MetricSpec(metricName="gpu-work-pending", targetSize=1)])
 
    def GetMetrics(self, req, ctx):
        # Normalize both signals to "replicas wanted", take the max —
        # the same OR-semantics KEDA gives multi-trigger ScaledObjects.
        import math
        wanted = max(max_gpu_util() / UTIL_THRESHOLD,
                     queue_depth() / QUEUE_THRESHOLD)
        return pb.GetMetricsResponse(metricValues=[
            pb.MetricValue(metricName="gpu-work-pending",
                           metricValue=int(math.ceil(wanted)))])

Deploy it where the GPUs are — a DaemonSet on GPU nodes with the same NVIDIA-container-runtime driver access the dcgm-exporter DaemonSet uses — and point a ScaledObject at it:

spec:
  scaleTargetRef: {name: vllm}
  minReplicaCount: 0
  maxReplicaCount: 8
  triggers:
    - type: external
      metadata:
        scalerAddress: keda-gpu-scaler.gpu-ops.svc:6000
        metricName: gpu-work-pending

Two things the sketch deliberately leaves visible. First, StreamIsActive is unimplemented, so KEDA falls back to polling IsActive every pollingInterval — fine at 15 seconds, and one less streaming RPC to get wrong. Second, multi-node fleets need aggregation: each DaemonSet instance sees only its own node, so you either scope one ScaledObject per GPU node pool or adopt an existing implementation — keda-gpu-scaler does exactly this NVML-direct pattern with max/min/avg/sum aggregation across devices, plus pre-built vLLM and Triton profiles. Build the sketch to learn the contract; adopt the project to run the fleet.

From pods to nodes: the scale-to-zero money story​

Pod-level scale-to-zero only saves money if the nodes follow. They can: the cluster autoscaler's Cluster API provider treats a MachineDeployment with a min-size-zero annotation as a node group that may vanish entirely, and recent autoscaler releases even handle DRA device claims when scaling from zero. Pods go to zero on idle, pending GPU pods summon a node, the node drains away after the idle timeout. The full loop is pending-to-Running in roughly a minute and a half on cloud GPU VMs, plus model-load time on top.

How much that loop is worth depends on your idle fraction. Take a representative hosted-GPU rate of about $0.53 per GPU-hour — roughly $380 a month per always-on inference pod:

Three caveats keep this table honest. First, bare-metal GPUs have effectively zero elasticity: the box bills 24/7 whether KEDA exists or not, so on owned hardware the achievable win is pods-to-zero on a pinned node (power and noisy-neighbor relief, not dollars). The dollar table above is a cloud-GPU-VM story.

Second, scale-up takes minutes — node boot plus multi-gigabyte weight download into VRAM — so latency-sensitive inference still wants minReplicaCount: 1 as a warm pool and scale-to-zero only for batch, evals, and dev sandboxes. Third, GPU scale-down that outruns its cooldown will thrash between zero and one, and every cycle pays the full cold-start price; set cooldownPeriod longer than your measured cold start, not shorter.

Where KEDA stops: replica counts are not device awareness​

KEDA answers exactly one question — how many pods — and never touches the question underneath it: which slice of which device each pod lands on. Two tenants sharing one GPU for inference sandboxes do not need a replica count; they need fractional, scheduler-visible device sharing with memory and compute enforcement, which is HAMi's whole job (nvidia.com/gpumem-style fractions the scheduler can bin-pack), or a DRA ResourceClaim once the driver ecosystem covers your cards.

A queue-depth metric cannot express cross-tenant fairness, and no threshold tuning will make it. The honest architecture is layered: KEDA for how many replicas the load deserves, the device scheduler for where each replica is allowed to sit. Conflating the two layers is how teams end up with perfect autoscaling over a single GPU that eight tenants fight over.

That layering is also the adoption order. KEDA's Prometheus scaler is a weekend project on top of monitoring you already run; the external scaler is the next weekend when latency matters; fractional sharing enters the picture the day a second tenant's sandbox needs the same card as the first — not before. Each layer pays for itself independently, which is exactly why "HPA can't see GPUs" was never a reason to buy a bigger scheduler. It was a reason to give the scheduler's neighbor better eyes.

Measure before you tune: record time-to-first-token from zero (node boot, image pull, weight load) and set cooldownPeriod above it, pick the utilization threshold where your p99 latency starts bending rather than a round number, and size maxReplicaCount by GPUs you actually own — KEDA will happily request replicas no node can schedule. Eyes first, then appetite, then discipline.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. When your agent sandboxes outgrow one GPU node, the same KEDA-plus-node-pool loop above is how a self-hosted fleet stops paying for idle silicon. Star the repo on GitHub or deploy your first app today.