Fork in 3ms: What Kedge's Snapshot Tree Means for the Next Agent Sandbox

A fresh Machine on Hetzner takes 45 seconds if you're lucky and twelve minutes if you're not. A Kedge fork takes 3 milliseconds. That gap is not an optimization — it's a different primitive.

Kedge, the July 2026 Show HN that pitched a full-stack cloud built on forkable VM snapshots, claims a hardware-isolated microVM forks from a warm-pool tree — kernel, then base runtime, then app layers — into a new sandbox or service instance in about 3ms, billed on actual CPU-milliseconds and resident memory rather than provisioned capacity. Whether or not you ever run Kedge itself, the architecture it demonstrates is the one every AI-agent sandbox discussion is converging on in 2026, and it exposes exactly what a plain Cluster-API-provisioned fleet doesn't give you for free.

Here is the number that reframes the rest of the post:

The bottom row is the promise. The rest of this post is what it costs to get there — and what it means for a fleet that already owns its machines.

What Kedge actually built​

Kedge's pitch is a snapshot tree, not a faster container.

The warm pool holds immutable memory images at each layer of the stack. At the root, a booted Linux kernel with drivers settled and init quiet. One level up, the base runtime — say Python 3.12 plus pip cache, or Node 22 plus node_modules baked. At the leaf, the app's own code and warmed caches. Each layer is a Firecracker memory snapshot created once and then forked N-way via copy-on-write. Daughters share the parent's memory pages until they write, so the next fork lands in a warm, ready environment instead of replaying a boot.

Three primitives combine to make the 3ms number real:

• Firecracker snapshot/restore with UFFD. The VM is not rebooted. Its memory and device state are restored from a snapshot, with lazy faulting via userfaultfd so only touched pages are materialized. The measured cost is the tens-of-milliseconds class: firepod's demo clocks 0.003s wall time for snapshot run, bhatti advertises a paused sandbox resuming and executing a command in under 3ms, and the mitos/forkd lineage reports P50 warm-claim activate around 27ms on a bare-metal reference node. The spread is real hardware and image-size dependent, but the order of magnitude is consistent.
• N-way CoW fork. A live parent VM can be branched into many children without re-snapshotting. Projekts like forkd advertise spawning 100 children in about 100ms from a warm parent and branching a live VM in about 150ms, with about 16 MiB resident overhead per sandbox and per-instance quotas around 5 MiB. The daughters diverge on write — the classic fork() semantic, applied to a whole machine.
• Warm-pool serving. The first boot of a layer pays the full cold cost and creates the snapshot. Every subsequent boot restores from it. Implementations like fcctl report cold boot around 1.5s falling to about 300ms via per-slot snapshot caching, and amber describes forking from a warm template in tens of milliseconds. The pool itself is the cache.

Billed on actual CPU-milliseconds, resident memory, and bytes stored rather than provisioned capacity, the model mirrors what SEUSS described for serverless in 2019 — a warm start of a no-op JS function in 3ms from a pool of immutable memory images — but re-applied to a full VM instead of a language isolate. Nothing is consumed when a unit is created from the snapshot. That is the whole trick.

Why 3ms changes the agent sandbox game​

Most tenant apps are not asking for 3ms. An HTTP service that scales on traffic is happy with KEDA or Sablier waking it in hundreds of milliseconds. An AI agent running a coding loop is a different workload shape.

A coding agent wants a disposable, pre-warmed environment per task. Open a repo, install deps, run tests, try a patch, throw it away — then do it again with a different prompt variation. If each attempt costs 1.5 seconds of cold boot plus 400ms of container startup, a 50-step agentic loop spends minutes just waiting for environments. At 3ms per fork, the environment cost disappears into the inference latency.

Three consequences follow directly:

• Branching replaces cloning. Instead of rebuilding state from scratch, the agent forks a live VM mid-session. One parent with the repo loaded and tests green becomes five children testing five patches in parallel, each isolated by KVM and each diverging only on the pages it touches. Daytona and E2B expose OCI workspaces and Firecracker microVMs, but neither advertises a true N-way CoW fork in the open-source path — Daytona's sandboxes clock under 90ms, E2B's resume path sits in the tens of milliseconds, and Fly.io Sprites restores in the single-digit seconds-to-milliseconds boundary depending on image warmth. The Kedge-shaped design makes branching a first-class primitive, not a template re-create.
• Per-task billing matches per-task shape. A fleet that bills per-server-hour amortizes idle time into every tenant. A sandbox layer that bills CPU-milliseconds only while code is executing maps directly onto bursty agent tasks that spend most of their wall time waiting on the model. The tax on idle goes to zero. The tax on the burst is exactly the burst.
• Isolation stays hardware-grade. V8 isolates achieve 5–11ms cold starts with process-level sharing, which is faster on paper, but they trade away the KVM boundary. A microVM fork keeps a real kernel per tenant task. For untrusted agent-generated code running arbitrary shell commands, that boundary is the one you want between a bad rm -rf / and the next tenant's workspace.

The market is pricing that boundary today: E2B around $0.05/vCPU-hour plus a $150 Pro tier, Fly.io Sprites at roughly $0.07/CPU-hour plus storage, Runloop and Daytona in the same band, with per-snapshot storage as a quiet extra. None of them yet offers open, self-hosted forking at the Kedge price point — which is precisely why a self-hosted fleet has to decide whether to build it.

What a plain Cluster API fleet gives you today​

A Cluster API fleet on Hetzner is excellent at what it was designed for: declarative lifecycle for machines and clusters. It is not, by itself, a snapshot-forking sandbox.

The provisioning path is well instrumented by 2026 issue trackers. A cloud VM provisioned via CAPH can land in 30 to 60 seconds end to end — select server, gRPC provision, boot Ubuntu Cloud, join. A bare-metal host is slower: CAPH reports routinely show 5 to 13 minutes of StillProvisioning before ProvisionSucceeded, and a few pathological retries stretch past two hours when the health check never finds a matching ProviderID. The autoscaler-driven scale-up lands in the 60 to 90 second range even on managed ASGs. All of this is fast for infra. It is glacial for an agent asking for an environment this second.

Above the machine, the container layer helps but does not close the gap. A container on a warm node starts in hundreds of milliseconds if the image is already present, or low seconds if it must be pulled. A KEDA ScaledObject handling 0-to-1 plus an HPA above it can wake an idle Deployment from zero, but the wake is still a pod scheduling and container start — not a memory-image fork. Sablier's reverse-proxy-intercept model delays waking until an actual request arrives, which is the right optimization for HTTP-idle savings, but it is still optimizing a cold start that a fork would never have paid.

In short: Cluster API gives you fleet operations. Kubernetes gives you scheduling. Neither gives you a snapshot tree below the Machine abstraction that can hand an agent a warm Linux VM in milliseconds.

What it would take to build on Hetzner​

Closing the gap means adding a layer that Cluster API does not currently own: a warm-pool and snapshot lifecycle that sits below Machine and above raw KVM.

Concretely, a Hetzner-backed fleet that wants Kedge-like forking has to build or integrate at least four pieces:

1. A snapshot-capable microVM runtime on each worker. Plain containerd and runc will not do. The fleet needs Firecracker (or Cloud Hypervisor / RustVMM equivalent) on every node that will serve forks, plus the UFFD plumbing for lazy page faulting and CoW sharing. Projects like bhatti, forkd, and firepod demonstrate the primitives on Latitude-style bare metal; the gap on a Hetzner fleet is wiring them into the node image that CAPH produces, not just running them on a pet machine.

2. A warm-pool controller. Snapshots at each tree layer — kernel, base runtime, app — must be built, versioned, and served. The controller's job is to hold those immutable images in memory, watch for invalidation (kernel bump, base image update), and re-snapshot on change. There is no existing Cluster API provider that does this; it is a new control loop either inside or alongside the fleet, reconciling desired pool state the way CAPH reconciles desired machine state.

3. A forking API and per-sandbox lifecycle. Agents do not want kubectl create pod. They want fork(snapshot_id) returning a hardware-isolated VM with a workspace, an egress policy, and a TTL-based teardown. The API must enforce quotas (16 MiB resident is the demonstrated ballpark, not the unlimited RAM of a full node), network egress filtering per sandbox, and ephemeral-namespace semantics so state does not leak between forks. The closest analog in the fleet today is Kata Containers or gVisor — both offer stronger isolation than runc, neither offers snapshot-tree forking out of the box.

4. Storage and billing bookkeeping. Snapshots live on disk. On Hetzner, bytes stored are already paid for in the flat node price, but the fleet still needs accounting for snapshot retention and GC — stale app-layer snapshots from rolled-back deploys can silently accumulate the way old container images do, only larger. Billing, if the platform exposes per-tenant sandbox usage, must meter CPU-milliseconds and resident memory per fork rather than charging for the underlying node — the same inversion Kedge bakes in.

None of this is exotic. Every piece has an open-source reference. The cost is integration and operational surface: another daemonset, another controller, another set of kernel compatibility constraints (Firecracker pins to specific KVM versions and host kernel features), and a new failure mode where the warm pool itself is the thing that needs warming after a node reboot.

The honest benchmark for "good enough" is not matching Kedge's 3ms. It is getting the P50 resume under 30ms and the fan-out to 100 sandboxes under half a second on fleet hardware — a 100x improvement over the current CAPI Machine path, with the remaining gap made up in product semantics (pre-warm the pool on node boot, fork on agent request) rather than raw speed.

The billing inversion​

Kedge's meter — CPU-milliseconds, resident-memory, bytes-stored — and a Hetzner fleet's flat per-server-hour are not two ways of charging for the same thing. They are opposite answers to "what is the unit of compute."

A Hetzner CX or CPX box is already sunk cost. Whether it runs zero tenants or fifty, the invoice is the same number every month. The platform's job is bin-packing to keep that fixed capacity full. Every idle minute is not billed — it is wasted capacity the operator already paid for.

A snapshot-fork meter only ever charges when code is executing or memory is resident. Idle is free by construction. For a bursty, mostly-idle workload like agent loops — five steps at 3.2x convex token cost to fifty steps at 30x, per the token-spend analyses making the rounds in 2026 — the metered model wins on cost alone, because most of the time the sandbox is waiting on the model, not burning CPU.

For a steady-traffic app, the math flips. A workload that is busy 80% of the wall clock pays for 80% of the wall clock on a meter but only 100% of a smaller fixed number on flat hardware. The breakeven is exactly where Leapcell and similar platforms draw their "switch to persistent server" line: once the sandbox is busy more than it is idle, owning the box is cheaper than renting CPU-milliseconds.

The decision for a fleet is not which meter is right globally. It is which workload you are actually pricing. Agent sandboxes want the meter. Long-lived tenant services want the box. A fleet that tries to price both on one meter will get one of them wrong.

Where the fleet still wins​

A snapshot tree does not replace a fleet. It sits alongside it.

Tenant services that are stateful, need persistent volumes, or talk to a Postgres instance pinned to a node gain nothing from being forked from a snapshot every request. They want the durable scheduling, rolling upgrades, and Machine lifecycle that Cluster API already gives — MachineDeployment rollouts, MachineHealthCheck remediation, cluster-class-driven upgrades via tools like k0rdent or plain CAPH. Forking is the wrong primitive for a database.

The fleet also brings the one thing a snapshot cloud inherently cannot: you own the machines. A vulnerability in Coolify's single-daemon model or a vendor outage in a hosted sandbox provider is not your vulnerability or your outage. On owned Hetzner hardware, the control plane is yours to patch, the warm pool is yours to keep warm, and the audit log distinguishes the agent that forked from the human that deployed. The self-hosted tax is real — you run the pool — but the blast radius is yours to bound.

For that reason, the most credible architecture is a split fleet: long-lived tenant services on Cluster-API-managed nodes with ordinary scheduling, plus a dedicated sandbox tier on Firecracker-equipped nodes behind a fork API. k0rdent's super-control-plane pattern and Kube's multi-cluster distribution primitives are natural homes for that split — a sandbox workload cluster managed by the same control plane as the tenant workload clusters, but with a different runtime class.

Forking a running VM in 3ms feels like magic until you see the warm pool doing the real work. The snapshot tree is a cache — immutable images, layered, CoW-shared — and like any cache it trades setup cost for hit latency. Kedge's contribution is not the millisecond number. It is the packaging: a full-stack cloud where the primitive is fork(), not create(), and the meter is CPU-milliseconds, not node-hours.

A Cluster API fleet does not get that primitive for free. But it gets the substrate that makes building it tractable — declarative machines, a homogeneous Hetzner node pool, and a control plane you operate. The gap between 90 seconds and 3 milliseconds is large in a table and small in architecture: a Firecracker runtime, a warm-pool controller, a forking API above them, and bookkeeping below. The question is no longer whether the pieces exist. They do. The question is which platform does the wiring first — the rented sandbox or the fleet that already owns the hardware.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources:

• Kedge Show HN (July 2026) — forkable VM snapshots, warm-pool tree, 3ms fork, CPU-ms/resident-mem/bytes-stored billing (per TODO specification)
• SEUSS: Rapid serverless deployment using environment snapshots (arXiv 1910.01558) — warm pool of immutable memory images, 3ms warm start for no-op JS function
• ejc3/firepod — instant VM cloning from warmed snapshot demo, 0.003s measured fork
• mesa-dot-dev/bhatti / bxxd/forked-bhatti — paused sandbox resumes and executes command in under 3ms; Firecracker microVM orchestrator
• prombot-ai/forkd — spawn 100 children in ~100ms from warm parent, BRANCH a live VM in ~150ms, per-instance overhead
• suryatmodulus/mitos / paperclipinc/sandbox — Firecracker snapshot restore in tens of ms, P50 ~27ms warm-claim activate, N-way CoW fork
• paradisecy/fcctl — sub-20ms warm starts via paused-VM resume, per-slot snapshot caching (300ms vs 1.5s cold)
• lupodevelop/amber — forks from warm template CoW, tens of ms handoff
• boringcomputers/nehemiah — snapshot restore ~3ms p50, 20 microVMs fan-out 282ms wall-clock
• agentstep/mvm — competitor pricing table June 2026: Fly Sprites $0.07/CPU-hr, E2B $0.05/vCPU-hr
• third-party-collaboration/forkd comparison table — CubeSandbox, Daytona (<90ms), E2B isolation/pricing comparison
• perevillega.com — I Built Yet Another Sandbox for AI Coding Agents — Fly.io Sprites ~$0.02/hr, E2B/Daytona usage-based pricing context
• syself/cluster-api-provider-hetzner issues 1661, 1664, 1492 — CAPH StillProvisioning timings 5–13 min bare-metal, cloud VM 30–60s
• Hetzner k3s / hetzner-k3s tooling — KVM provisioning flow and cluster-creation context