Dora NodaOn September 1, 2026, DigitalOcean took its Spegel-based peer-to-peer OCI registry plugin for DOKS from public preview to general availability. The pitch is simple: every node in your cluster already stores the image layers it has pulled, so why should node number eleven download the same 800 MB tenant image from the internet when ten warm peers sit one LAN hop away? With the plugin on, it doesn't — containerd asks the cluster first and only phones home to the origin registry on a miss.
This is the kind of feature that sounds like free performance until you ask what it actually saves, in seconds, gigabytes, and registry pulls — and what it costs to run yourself on machines you own. This post works all of that out: the measured benchmarks, the scaling math for a small Cluster API fleet, an honest comparison against the pull-through cache you might already run, and the peer-discovery gotcha that bites self-hosters.
How Spegel works in 60 seconds
Spegel is an open-source, stateless OCI registry mirror. It runs as a DaemonSet — one pod per node — and each instance advertises the image layers its node already holds in containerd via a distributed hash table (DHT). A small mirror entry in containerd's configuration (a hosts.toml snippet per registry) routes pulls through the local Spegel instance first. Peer hit: layers stream node-to-node over the cluster network. Miss — nobody has the layer yet, or the 20 ms DHT lookup times out: containerd falls back to the upstream registry exactly as before.
Two properties make this design unusually safe to adopt. First, it is transparent: no image references change, no registry URLs get rewritten, no sidecar touches your manifests. Second, the fallback is the old behavior, so Spegel adds no new failure mode. Crusoe validated this the hard way on Crusoe Managed Kubernetes by killing every Spegel pod mid-pull during an active 8.5 GB image pull — the pull finished successfully from upstream with no intervention. Worst case, you are back to today's default behavior.
DOKS packages exactly this: enable the plugin and you get a k8s-spegel DaemonSet in kube-system, managed for you. Self-hosters get the same binary via Helm with one extra responsibility — the containerd mirror config, which is where the gotchas live (more on that below).
The numbers, honestly
Peer-to-peer pull stories always come with a big multiplier attached — "7x faster second-node pulls" is the genre — so let's start by asking 7x of what? The answer is always the transfer phase only: warm peer over a fast LAN versus the same bytes over a thin internet uplink. That ratio is real, and it is also the least interesting number, because end-to-end pull time includes unpacking, and unpacking doesn't care where the bytes came from. The honest benchmark reports both. Crusoe's July 2026 benchmark on a 56-node GPU cluster does exactly that, across three image sizes, with the headline result on a 17 GB NVIDIA NeMo image:
| Phase (17 GB image, warm cluster) | Without Spegel | With Spegel | Delta |
|---|---|---|---|
| Network download | ~165 s | ~74 s | ~55% faster |
| End-to-end pull (median) | ~348 s | ~307 s | ~12% faster |
| End-to-end pull (P99) | baseline | −17% | slowest nodes converge to median |
The download phase more than halved, but end-to-end moved only 12% because unpacking a 17 GB image takes roughly five minutes regardless — and download overlaps unpack, so the phases aren't additive. The P99 improvement arguably matters more than the median: P2P pulls drag the stragglers — the nodes with the worst path to the registry — back in line with the rest of the fleet.
For smaller images the transfer-phase win shrinks further in absolute terms while the cluster-wide savings stay exactly proportional. That second part is pure arithmetic, not a benchmark, so we can work it precisely. Rolling one image to N fresh nodes costs:
| Cost axis | Direct upstream pulls | Pull-through cache | Spegel P2P |
|---|---|---|---|
| Upstream egress | N × image size | 1 × image size | 1 × image size |
| Registry pulls consumed | N | 1 | 1 |
| Per-node transfer wait | full upstream transfer | LAN-speed transfer | LAN-speed transfer (warm peers) |
| New infra to operate | none | cache host + storage + GC policy | DaemonSet + containerd config |
Widen that across fleet sizes and image sizes and the shape is clear:
| Rollout | Direct: egress / pulls | Spegel or cache: egress / pulls |
|---|---|---|
| 50 MB utility image → 3 nodes | 150 MB / 3 | 50 MB / 1 |
| 800 MB tenant app → 10 nodes | 8 GB / 10 | 800 MB / 1 |
| 800 MB tenant app → 50 nodes | 40 GB / 50 | 800 MB / 1 |
| 17 GB ML image → 50 nodes | 850 GB / 50 | 17 GB / 1 |
Two observations. First, the per-node latency win ranges from negligible (sub-second on a 50 MB image) to minutes (a minute-plus of transfer on a 17 GB image) — the multiplier genre always quotes the rightmost column. Second, the egress and pull-count savings are identical for a pull-through cache and for Spegel, and they scale with N regardless of image size. On the pull-count axis this connects directly to Docker Hub's budget: 100 pulls per 6 hours anonymous (per source IP), 200 per 6 hours for a free authenticated account. A 10-node fleet behind one NAT IP burns 10 pulls per full rollout; fifteen rollouts in six hours puts an anonymous fleet at 150 against a 100-pull cap — past it — while the same fleet on Spegel or a cache spends 15. That is the quiet way small fleets hit rate limits: not one big image, but many nodes × many deploys sharing one IP budget.
The last row of the honest table is the outage row. Direct pulls: a ghcr.io or Docker Hub incident stops every node that hasn't cached the tag. Pull-through cache: serves whatever it holds until TTL or garbage collection evicts it, then fails too. Spegel: any node can serve the layers its peers hold, so rolling restarts and scale-ups of already-deployed images keep working mid-outage. Note the shared limit: nobody can serve a tag no node has ever pulled. "Registry-outage immunity" means immunity for cached content, not immunity from needing a registry.
What DOKS GA actually gives you
On DOKS the whole thing is one toggle, available on clusters running Kubernetes 1.36 or later (the plugin is disabled by default):
doctl kubernetes cluster update example-cluster --enable-peer-to-peer-oci-registry-plugin=trueThe same p2p_oci_registry_plugin object exists in the API and as a Terraform block, and you verify it the Kubernetes-native way:
kubectl get daemonset k8s-spegel -n kube-systemDESIRED should equal your node count, with one Running pod per node under the c3.doks.digitalocean.com/component=p2p-oci-registry label. That is the entire operational surface DigitalOcean exposes: no mirror URLs to configure, no storage to size, no cache-warming job to schedule. The preview-to-GA arc (July to September 2026) plus Crusoe shipping the same layer on its managed Kubernetes suggests P2P pulls are crossing from homelab trick to default managed-control-plane furniture.
Spegel vs. the alternatives, quantified
The table in the previous section already settled the core question: on egress and pull-count, Spegel and a pull-through cache tie — both reduce N upstream fetches to one. So the choice between them (and the heavier P2P systems) turns on operations, not bytes:
| Approach | Extra components | Transparent? | Failure behavior | Sweet spot |
|---|---|---|---|---|
| Pull-through cache (registry:2, Harbor proxy, zot) | cache host + disk + GC | yes | cache down = fall back to upstream (if configured) | small fleets that want one throat to choke and already run a registry |
| Spegel | DaemonSet only, no storage | yes | any peer down = fetch from another peer or upstream | fleets reusing the same images across many nodes, no new stateful box |
| Dragonfly | scheduler + manager + per-node daemon | yes | tracker outage degrades to upstream | large clusters where a dedicated distribution tier is worth staffing |
| Uber Kraken | tracker + origin + agent per node | no — image URLs must point at Kraken | tracker down = pulls fail unless rerouted | extreme scale (Kraken's reference point is 3 GB to 2,600 hosts in p50 ~10 s) |
Read the bottom row as the ceiling, not the competition: below roughly a thousand hosts, the origin-egress cost that justifies Kraken-style tracker infrastructure is noise, and the per-host RTT dominates — which is exactly the regime where "ask a warm peer first, else upstream" captures nearly all the available win for one DaemonSet. Dragonfly sits between: genuinely better fan-out at hundreds of nodes, but you operate a control plane for your image pulls. For a bex-sized fleet — a handful to a few dozen Hetzner machines pulling the same tenant images on every node — the realistic shortlist is Spegel versus the pull-through cache, and the tiebreak is whether you'd rather operate a stateful cache box with a garbage-collection policy or a stateless DaemonSet with a DHT.
Self-hosting it on your own fleet
Running Spegel outside DOKS is one Helm command plus one config file, and the config file is the part people get wrong. The checklist:
- Deploy the DaemonSet. Upstream Helm chart or the Talos/K3s-native paths — K3s embeds Spegel, and SideroLabs publishes a Talos guide. One pod per node, no persistent storage.
- Write the containerd mirror config. This is the gotcha: without a
hosts.tomlmirror entry (orregistries.yamlon K3s) pointing each registry at the local Spegel instance, containerd never consults Spegel and everything silently pulls upstream as before. A homelab PR that deployed only the DaemonSet documented exactly this failure — green pods, zero peer traffic. Verify with a pull on a warm second node and confirm upstream didn't see it. - Mirror order composes. Where Talos already writes a base mirror list, Spegel's entry is prepended, giving Spegel peers → pull-through cache → upstream. The two approaches stack rather than compete.
- Mind peer discovery across networks. On a single LAN, multicast-based discovery just works. Across Hetzner private networks or any routed topology, peers need routable addresses to find and serve each other — confirm the DHT bootstrap sees every node's reachable IP, not a link-local or pod-network address that other nodes can't dial. Gardener's Spegel integration solves the equivalent problem with a dedicated bootstrapper handing out bootstrap peers; on a static fleet, the Helm values for advertised address are the thing to get right.
- Keep the fallback. Leave
skipFallbackoff (the default) so an all-peers-down event degrades to normal upstream pulls — the behavior Crusoe verified by killing every Spegel pod mid-pull. Spegel v0.1.0 also switched to mirroring all registries by default; if you pin older charts, check which registries yours actually covers.
None of this is exotic, but step 4 is the one that pages you at 2 a.m.: partial peer visibility looks like flaky slow pulls on some nodes, not a clean error.
Where it doesn't help
Close with the honest limits, because the multiplier genre never does. The first pull of a new tag always hits upstream — P2P only amortizes the second through Nth. Small images are scheduling- and unpack-bound; saving half a second of transfer on a 50 MB image is unmeasurable in deploy latency. Single-node clusters gain nothing (there are no peers). And peer bandwidth isn't free: on bandwidth-metered VPS links, node-to-node transfers still count against something — they're just cheaper and faster than the same bytes from another continent. Size the expectation by your image-size histogram and node count, not by anyone's headline multiplier.
The real story of the DOKS GA is therefore not a speed number. It's that the "every node for itself" pull model — N identical downloads of N identical layers — is finally getting a boring, default, managed answer, built on an open-source component you can also run yourself with one DaemonSet and one config file. For a self-hosted PaaS that deploys the same tenant images to every machine in the fleet, that is the rare optimization that cuts latency, egress, and rate-limit exposure at once, with a fallback that is simply the status quo.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
