Dora NodaYour tenant's database p99 is decided long before your query planner runs. It is decided in the storage data path — the unglamorous chain of daemons, kernel modules, and network protocols between your Postgres pod and the actual NVMe cells holding its pages. On a self-hosted Kubernetes fleet, that chain has a name most operators never chose deliberately: Longhorn's V1 engine, speaking iSCSI through the Linux kernel. Published latency surveys put iSCSI-backed Kubernetes volumes at roughly 500 to 800 microseconds per operation. The NVMe-over-TCP path that Longhorn's V2 engine uses instead typically lands around 300 to 500, with tuned setups dipping under 200.
So here is the verdict up front. V2's kernel-bypass design buys a real, measurable tail-latency win for exactly the workload a self-hosted PaaS cares about most — tenant databases on owned NVMe. But it charges a fixed per-node rent of one full CPU core and 2 GiB of hugepages, demands raw block disks and specific kernel modules, and carries a still-maturing operational track record with failure modes V1 settled years ago. Most fleets should pilot V2 on a dedicated storage class this quarter, not cut the default class over.
The rest of this post is the receipt behind that verdict: where V1's microseconds go, what V2 changes mechanically, what the benchmarks actually say (including where the win disappears), the per-node price tag, what still bites, and a decision matrix for your fleet.
Where V1's microseconds go
Longhorn V1's data path is a product of its era. When a pod writes to a V1 volume, the I/O travels from the container through a kernel iSCSI initiator, across the network to a Longhorn engine process, down through the kernel storage stack again on the replica node, and finally to disk — with reads fanning out across replicas for good measure. Every one of those hops crosses the kernel boundary: a context switch out of userspace, queueing inside the kernel's SCSI and block layers, an interrupt-driven completion, and a context switch back.
None of those steps is slow in isolation. The tax is the accumulation — hundreds of microseconds of fixed overhead per operation that no amount of faster NVMe underneath can remove, because the bottleneck was never the media. It is the round trip through software layers designed in an era when disks were measured in milliseconds and a few hundred microseconds of stack overhead rounded to zero. On modern NVMe, where the device itself answers in tens of microseconds, the kernel path is the dominant term in the latency equation.
This is also why V1 performance complaints have a recognizable shape: throughput looks fine (bandwidth parallelizes), but p99 latency under concurrent load disappoints, and small random I/O — the exact pattern of a busy Postgres or Redis — suffers most. If your tenants' databases feel slower than the bare-metal specs say they should, the iSCSI data path is the first suspect, not the disks.
How V2's kernel bypass works, and what it measures
V2 rebuilds the data path around two ideas from the Storage Performance Development Kit (SPDK): move storage logic into userspace, and poll instead of interrupt. The spdk_tgt process running on each storage node owns its CPU core outright, spinning in a tight loop checking for completed I/O rather than sleeping and waiting to be woken. No context switches on the hot path, no interrupts, no kernel SCSI layer. Volumes are exposed to nodes over NVMe-over-TCP (with a ublk frontend option on newer kernels), so the initiator side speaks the NVMe protocol end to end instead of translating down to SCSI.
The measured results match the theory, with one important asterisk per measurement. Longhorn's own performance investigation wiki walks the full ladder — iSCSI with the V1 engine, then ublk with V1, then NVMe-over-Fabrics with V2 — and each rung is a large step up except on pure bandwidth. Community benchmarks on NVMe hardware report roughly 1.8 times the IOPS at roughly a fifth of the latency going from V1 to V2, which lines up with the industry rule of thumb that iSCSI-backed Kubernetes volumes sit at 500 to 800 microseconds while NVMe/TCP-backed ones sit at 300 to 500. The project's official KBench runs (three-node clusters on Equinix bare metal and AWS NVMe instances) show the same direction across IOPS, bandwidth, and latency charts.
Now the sensitivities, because this is where pilots get surprised:
- The win assumes NVMe. SPDK runs on any block hardware Linux supports, but on SATA SSDs the gains shrink and on spinning disks they mostly vanish — the media, not the stack, is the bottleneck there. Do not adopt V2 to rescue slow disks; it accelerates the stack above fast ones.
- Bandwidth caps out. In Longhorn's own investigation, NVMe-over-Fabrics with V2 hit a read-bandwidth ceiling around 2,321 MiB/s, and the ublk frontend beat it by up to 37 percent on some metrics. If your workload is large sequential reads, frontend choice matters more than engine choice.
- Replicas and network still dominate throughput. Every Longhorn write is replicated over your pod network, so inter-node bandwidth (7.9 to 15 Gbit/s in the official test setups) and replica count bound total throughput no matter which engine you run. V2 lowers per-operation latency; it does not repeal physics on your wire.
In short: V2 is a latency play for small random I/O on NVMe — databases, queues, latency-sensitive key-value stores — not a bandwidth play and not a slow-disk play.
The per-node price tag
Kernel bypass is not free; it relocates cost from latency to reserved resources. Every node that runs V2 volumes owes the following, whether or not any volume is mounted at the moment:
| Line item | What V2 needs | What V1 needed |
|---|---|---|
| CPU | 1 full core per node, spinning at 100 percent in spdk_tgt poll mode | Shared, on demand |
| Memory | 2 GiB locked as hugepages (1024 2MiB pages) per node | None reserved |
| Disks | Raw, unformatted block device registered as a block-type disk — a filesystem path will not do | Any mounted path, including the root disk |
| Kernel modules | vfio_pci, uio_pci_generic, nvme-tcp loaded on every storage node | open-iscsi and iscsid |
| Kernel version | 6.0 or newer for the ublk frontend; note a known SPDK memory-corruption risk on kernels 6.1 through 6.6 | No special requirement |
On a three-worker fleet of 4-core boxes, that CPU line alone is a quarter of a worker reserved for storage polling — trivially worth it when the alternative is tenant databases missing their latency SLOs, and pure waste on a fleet whose volumes hold mostly static assets. The disk requirement is the real planning constraint: V2 cannot share the OS disk the way V1's default path does, so each node needs a second device or a carved partition before V2 schedules a single replica.
Before committing hardware, run the project's own preflight check, which validates CPU features, hugepages, modules, and dependencies in one pass:
longhornctl check preflight --enable-spdkOn Talos Linux — the common OS under Cluster API fleets on Hetzner — the prerequisites translate to machine config: a vm.nr_hugepages: "1024" sysctl plus the kernel modules on every worker class, applied with a rolling reboot. Operators who have done it recommend adding the V2 storage class as inert first (provisioning nothing until the engine is enabled and a block disk is registered), because enabling the engine while volumes are attached is blocked — the cutover needs a maintenance window, not a config push.
What still bites
This is the section that keeps the verdict at "pilot" rather than "migrate." V2 has been climbing out of experimental status across the 1.6 through 1.12 release lines (1.12.1 is current as of this writing, with 1.11 shipping February 2026), and the sharp edges are specific enough to name:
- No live engine upgrades. V2 volumes must be detached before an engine upgrade — plan maintenance windows around Longhorn upgrades, not just Kubernetes ones.
- No mixing engines per volume. A volume's replicas are all V1 or all V2, so migration means provisioning fresh V2 volumes and moving data. (Since 1.6 you can at least back up on one engine and restore on the other, which is the sanctioned crossing.)
- Partial feature parity. Check the project's V1/V2 parity wiki before assuming a V1 feature exists on V2 — the V2 backing-image support was deprecated outright in 1.11, and volume-granular settings parity is still an open feature request.
- Attach latency at scale. The release notes flag growing V2 attach latency as volume counts climb, suspected in NVMe-TCP connection handling — exactly the failure mode that appears after the pilot succeeds and the tenth tenant onboards.
- Real-world scar tissue. Self-hosters running Talos on Hetzner-class hardware have publicly retired V2 after it failed under full disaster-recovery restore load (open 1.12 issues, tracked as #13314 and #13315), and others have disabled an enabled-but-unused V2 engine just to reclaim the 2 GiB of hugepages per node. These are not "V2 is broken" stories; they are "V2's worst day arrives later than V1's" stories, which is precisely what a tenant-facing platform must budget for.
None of these is a veto. All of them are arguments against making V2 the default storage class before you have operated it through an upgrade, a restore, and a noisy-neighbor incident of your own.
The decision: pilot, adopt, or stay on V1
Put workload, hardware, and fleet shape together and the answer is usually one row of this matrix:
| Your situation | Call |
|---|---|
| Tenant Postgres/Redis/valkey on NVMe, latency SLOs that tenants notice | Pilot V2 now on a dedicated storage class; move one latency-sensitive tenant and compare p99 for a full release cycle |
| Bulk file storage, backups, image registries, mostly sequential I/O | Stay on V1 — V2's win is latency on random I/O, and the per-node rent buys you nothing here |
| SATA SSDs or spinning disks under the fleet | Stay on V1 — fix the media first; SPDK cannot bypass a slow device |
| Fewer than ~3 storage nodes or volumes that mostly sit idle | Stay on V1 — 1 core plus 2 GiB per node is a heavy fixed tax on a small fleet |
| Already running V1 everywhere with healthy headroom | Do not migrate in place; stand V2 alongside and let new latency-sensitive volumes land there |
If the matrix says pilot, the rollout that has worked for others is deliberately boring: run preflight, add block disks to one worker class, enable the engine in a maintenance window, create a non-default V2 storage class, move one tenant's database, and keep V1 as the default until V2 has survived your upgrade, your restore drill, and your worst noisy-neighbor week. On the way, confirm your kernel situation (6.0 or newer for ublk, and know where you stand on the 6.1-through-6.6 SPDK caveat) rather than discovering it mid-migration.
The deeper point is architectural, and it is why this matters beyond one storage release. Owning your hardware means owning the storage layer — there is no cloud block-storage API to rent your way out of the problem. Longhorn V1 won the small-fleet default because it runs on anything, including the OS disk. V2 is the first honest attempt at making that owned storage fast, priced in reserved cores and hugepages instead of dollars per GB-month. That is a trade a self-hosted PaaS is built to evaluate on its own terms — measure your p99, count your cores, and let the numbers vote.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
