vLLM vs Ollama in Production: What PagedAttention's 19x Throughput Gap Really Buys on Owned GPUs

On the same GPU, the same model, and the same prompt set, vLLM hit 793 tokens per second while Ollama flatlined at 41. That is a 19x gap — not a rounding error, not a warm-up artifact, and not a trick of quantization.

The same pair of engines, measured at one concurrent user, land within 20% of each other.

Both facts are true. They look contradictory only if you treat "which inference engine is faster" as a single number instead of a curve that depends on the one variable teams forget to ask about: how many tenants are hitting the GPU at the same time.

That question — not model size, not prompt length, not GPU SKU — is the variable that decides whether a Cluster-API fleet's GPU node pool wants vLLM's PagedAttention engine or Ollama's single-user simplicity. The engines are not interchangeable runtimes with a speed ranking. They are tools built for different concurrency regimes that happen to speak the same OpenAI-compatible API. Picking the wrong one for your tenant count costs you real money on owned hardware, where the GPU is a fixed monthly cost and wasted utilization is not someone else's metering problem to hide behind.

This post puts the full curve on one page: what PagedAttention actually does, where the 19x gap opens and where it closes, what vLLM ships for production that Ollama doesn't, where Ollama still wins, and what each choice costs month-over-month on a Hetzner GPU box under Cluster API.

The 19x number and exactly where it comes from​

Red Hat's 2026 serving benchmark ran Qwen2.5-7B-Instruct on a single NVIDIA T4, one of the cheapest data-center GPUs still sold, and hammered both engines with rising concurrency until each engine's throughput stopped climbing.

vLLM scaled. Ollama didn't.

The 19x peak is the top-right corner of the chart: 128 concurrent requests, each generating dozens of tokens, measured as aggregate tokens per second coming off the card. A second benchmark on an H100 with the same pattern found an 8-9x gap — smaller in absolute multiple because the bigger card raises Ollama's own ceiling, but structurally identical. The July 2026 Towards AI synthesis of the same Red Hat data reported P99 latency at peak load as 80 ms for vLLM against 673 ms for Ollama on that T4. A separate SitePoint April 2026 run at 50 concurrent users measured vLLM at ~6x throughput with P99 under 3 seconds while Ollama's P99 hit 24.7 seconds.

The takeaway is not "vLLM is 19x faster." The takeaway is "vLLM's throughput curve is a slope, Ollama's is a plateau" — and the plateau starts early.

What PagedAttention actually does​

The reason that slope exists is not a better kernel or a faster tokenizer. It is a memory-management idea borrowed from operating systems — paging — applied to the one data structure that determines how many requests a GPU can serve at once: the KV cache.

During autoregressive generation, every token a model has seen (prompt plus everything generated so far) leaves behind a key vector and a value vector. Those vectors are the KV cache. On a 7B model at 2K context, the cache is already hundreds of megabytes per request and it grows with every token generated. Traditional runtimes allocate that cache as a single contiguous block per request, sized for the maximum sequence length up front. That wastes two things at once: the unused tail of every allocation (internal fragmentation) and the gaps between allocations that cannot be reused for another request (external fragmentation). At 32 concurrent requests the GPU has plenty of compute left but nowhere to put request 33's cache without evicting something.

PagedAttention, introduced by Kwon et al. at UC Berkeley's Sky Computing Lab (SOSP 2023, paper 2309.06180), splits each request's KV cache into fixed-size blocks — typically 16 tokens per block — and stores a block table that maps logical block positions to whatever physical pages happen to be free. Blocks do not need to be contiguous. Allocation is on demand: a request only takes the pages it has actually filled, and when generation finishes its pages return to a free pool immediately. Crucially, two requests that share a prefix (the same system prompt, the same few-shot examples) can point their block tables at the same physical pages instead of duplicating them.

The effect is near-zero waste in KV-cache memory and the ability to share cache across requests without copying. Berkeley's original evaluation reported 2-4x throughput improvement over FasterTransformer and Orca at the same latency target. The 2026 multi-engine benchmarks that produce the 19x number are measuring the same mechanism under heavier concurrency than Berkeley's 2023 setup, on longer generations where fragmentation hurts more.

Continuous batching is the companion trick. Instead of waiting for a whole batch of requests to finish before admitting the next one (static batching), vLLM's scheduler inserts new requests into the running batch as soon as a slot frees, token by token. PagedAttention makes that insertion cheap — there is always a free page for the newcomer's first block — while Ollama's llama.cpp-derived queue handles requests sequentially and only parallelizes within a single request via threading.

None of this matters at one concurrent user. At one request, there is no fragmentation to avoid, no prefix to share, and no queue to fill. That is why the gap collapses to ~20% in that regime.

The full curve: throughput and latency by concurrency​

The table below synthesizes the Red Hat T4 run, the H100 8-9x corroboration, the Towards AI latency cut, and the SitePoint P99 numbers into a single shape. Treat the absolute tok/s values as order-of-magnitude on a T4 with a 7B model — your numbers will shift with model size, quantization, and GPU SKU — but the curve shape is consistent across every 2026 benchmark this post drew from.

* vLLM's P99 at 128 includes TTFT (time to first token) variance from continuous batching's admission policy; the Towards AI cut reported ~80 ms at peak aggregated throughput.

Three things to read off the curve:

• The gap is a step, not a slope. Ollama goes flat by 5-8 concurrent. The advantage is not that vLLM gets faster with more users — it is that Ollama stops scaling almost immediately.
• Latency tells the same story harder. At 50+ concurrent, Ollama's P99 is an order of magnitude worse. That is what "sequential queue" feels like to the 50th user waiting for the first 49 to finish.
• At low concurrency, flip the comparison. If your fleet serves one tenant's batch job or a single developer's chat loop, the 19x number is the wrong number to optimize for. The right one is the 1.2x gap — where setup cost and operability dominate.

What vLLM ships for production that Ollama doesn't​

Throughput is only half the fleet question. The other half is what happens after you put an engine on a Kubernetes node and need to operate it for a month without babysitting.

This is where the two projects' origins show. Ollama started as a model manager for local development: ollama pull llama3 and a daemon that Just Works on a Mac or a Linux box with or without a GPU. vLLM started as a serving system for production clusters. The difference is visible in a short checklist that maps directly to the page a fleet operator actually configures.

Two rows deserve a longer note.

Metrics and autoscaling. On a fleet, a pod that reports 8% CPU while its GPU sits at 100% is not "under-utilized" — it is saturated on the resource the HPA cannot see. The standard fix is a DCGM exporter → Prometheus → KEDA pipeline, but the vLLM operator community and projects like keda-gpu-scaler now expose queue length and GPU-cache pressure directly as Prometheus gauges that KEDA can scale on with 2-4 second latency, bypassing the slower DCGM path. Ollama has no equivalent GPU-aware autoscaling surface — which is correct for its use case, because a single-user runtime should not need one.

Probes that mean something. A vLLM pod that has not yet loaded its weights into GPU memory should not receive traffic. Its readiness probe reflects that. An Ollama daemon reports readiness as soon as the process is up, which is fine on a laptop and wrong on a fleet behind a Service that will route a user's first request into a cold model and return an error.

Neither checklist makes one engine "better." It makes them suited to different deployment targets. A production fleet needs the first column. A developer laptop needs the second.

Where Ollama still wins — and it wins clearly​

Throughput benchmarks are easy to read as "bigger bar wins." At low concurrency the right reading is the opposite.

Single-user parity. At one concurrent request, Ollama is within ~20% of vLLM on the same GPU, and on a smaller model or a quantized GGUF it can be faster due to less scheduling overhead. If your workload is one developer running a coding assistant against a local 7B, the 19x number never enters the picture.

CPU fallback. Ollama runs without a GPU at all. That is not a curiosity — it is the reason a team can give every developer a local model on their laptop and only call the fleet's GPU pool when a job actually needs it. vLLM has no CPU path.

Setup cost. curl -fsSL https://ollama.com/install.sh | sh && ollama run llama3.1:8b versus a Helm chart with GPU resource claims, node selectors, taints and tolerations, and a weights pre-download Job. The second setup is correct for a fleet. The first is correct for getting a new hire productive in ten minutes. A 2026 cost comparison of developer-local inference found Ollama cut inference cost by roughly 70% for genuinely single-tenant, low-concurrency use compared to calling a hosted API per token — with no GPU infrastructure to operate at all.

Single-model focus without fragmentation. If the workload is genuinely one model serving one tenant on one card — a per-customer fine-tune, a personal agent, a background batch that processes one document at a time — there is no shared prefix to deduplicate and no concurrent cache pressure to page around. Ollama's simpler execution path is not wasting anything in that regime; vLLM's machinery is idle.

The rule of thumb from every benchmark's crossover analysis is the same: below ~5 concurrent users, pick on operability; above ~5, pick on throughput. At five concurrent the measured gap is already several multiples, and it only widens from there.

What each engine costs on owned hardware​

On a self-hosted fleet the GPU is a fixed monthly cost — a Hetzner GPU line or an owned bare-metal card that you pay for whether it serves one request or one hundred. The right cost question is therefore not "dollars per token at peak efficiency" but "how many tokens per second does this fixed-cost asset produce under each engine at the concurrency your tenants actually send."

The owned-hardware row is where the 16-19x gap becomes a direct cost multiplier: at 32 concurrent, the same fixed-cost card produces roughly 16x more tokens per hour under vLLM. That throughput translates one-to-one into lower amortized cost per million tokens, or into fewer cards needed to hit a throughput target. At 1 concurrent the two engines produce essentially the same tokens per hour, so the cost ratio collapses to ~1.2x and the operational simplicity of Ollama can outweigh the throughput difference.

For a fleet operator planning capacity, the practical move is to bin tenants by concurrency, not by total token volume. A fleet that serves many low-concurrency tenants (personal agents, per-developer sandboxes) can mix both engines: vLLM on the multi-tenant GPU pool, Ollama on per-tenant CPU or small-GPU sandboxes where isolation matters more than packing density.

The decision guide​

If you only take one figure from this post, take this one. Let concurrency — the number you can actually measure from your gateway logs — choose the engine.

The sensitivity to check: if you cannot estimate concurrent requests per GPU, instrument vllm:num_requests_waiting or your gateway's queue depth for a week before committing hardware. The wrong estimate is expensive in opposite directions — buying vLLM operability for a fleet that never exceeds 2 concurrent pays in complexity you don't need; running Ollama for a fleet that peaks at 50 concurrent pays in P99 latency your users feel on every request.

Running your pick on a Bex fleet​

A self-hosted PaaS that provisions its own machines has a freedom hosted GPU rentals don't: the node pool is not a SKU you rent by the hour but a Cluster API MachineDeployment you declare once and let the fleet reconcile.

For vLLM that looks like a GPU node pool behind a Deployment with a GPU resource claim, a readiness probe that gates on model residency, and a KEDA ScaledObject keyed on vllm:num_requests_waiting rather than CPU. With Kubernetes 1.34's DRA graduated to GA, the same card can be fractionally claimed by multiple agent sandboxes — structured ResourceClaim objects with topology-aware placement, rather than the old nvidia.com/gpu: 1 integer-count hack that forced one pod per card. That is the primitive that turns "one H100 per tenant" into "one H100 per dozen low-concurrency tenants packed with PagedAttention underneath."

For Ollama the shape is simpler and deliberately so: a CPU or small-GPU pool, no DCGM sidecar, no KEDA external scaler, a Deployment that tolerates spot or preemptible nodes because a single-tenant sandbox restarting is cheap. Some fleets run both shapes side by side — vLLM on the throughput-optimized GPU pool for the multi-tenant hot path, Ollama on the isolation-optimized pool for per-tenant sandboxes that must not share a GPU at all.

The 19x number is not the whole decision. It is the top-right corner of a curve that also contains a near-parity point at the bottom left. Own the curve, and the engine choice becomes a capacity-planning input — concurrency in, dollars per million tokens out — rather than a benchmark trophy you chose on brand.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. The same Cluster API fleet that runs your web services can provision the GPU node pool above — same bex.yml, same Render-compatible API, same git push whether the workload is an HTTP service or an inference endpoint. Star the repo on GitHub or ship your first GPU-backed service today.