TurboFieldfare Fits a 26B MoE Model in 2GB of RAM: What SSD-Streamed Experts Mean for Your Cheapest Inference Node

A 26-billion-parameter model now runs in about 2GB of RAM — not on a GPU server, but on an 8GB MacBook Air. The trick is embarrassingly legible in hindsight: in a Mixture-of-Experts model only a small fraction of weights fires per token, so keep the shared core resident and stream the experts off SSD as the router calls for them. Throughput becomes a storage-bandwidth problem, not a FLOPS problem: roughly 2GB of weights touched per token, an LFU cache absorbing the hot experts, and whatever miss traffic remains must cross the SSD bus every single token. On a laptop SSD that buys 5–6 tokens per second; on faster storage, 31–35.

That is the sentence this whole post exists to unpack: MoE inference has a cheat code, and the cheat code is priced in GB/s, not in VRAM. Here is what TurboFieldfare actually demonstrated, the per-token byte math that explains its benchmark numbers, and what it means for sizing the cheapest inference node on a self-hosted fleet — a node where the NVMe drive is the spec that matters and the RAM tier is nearly irrelevant.

How TurboFieldfare actually works​

TurboFieldfare is a custom Swift and Metal runtime by iOS engineer Andrey Mikhaylov, open-sourced under Apache 2.0 at drumih/turbo-fieldfare. It hit the Hacker News front page as a Show HN in late July 2026, pulling roughly 640 points and over 220 comments. It runs one model: the instruction-tuned Gemma 4 26B-A4B, Google DeepMind's April 2026 open MoE release (Apache 2.0, LMArena 1441). The model card is the whole ballgame, so here it is:

Two design choices do all the work. First, the split: everything the model touches on every token — attention, the shared expert, embeddings, the router — stays in memory as a 1.35GB core, while the 128 routed experts per layer live on SSD in 4-bit and get paged in only when the router selects them. Second, an LFU cache over the expert pool, so frequently-picked experts stop costing I/O at all. The runtime is deliberately model-specific rather than a wrapper around MLX or llama.cpp, which is what let the author co-tune kernels, cache policy, and I/O scheduling against one architecture — the repo ships a curated inventory of 103 measured experiments across kernels, caching, I/O, prefill, and decode.

The part fleet operators should notice: this is a throughput story told entirely in storage terms. The M2 Air and the M5 Pro differ enormously in storage bandwidth and page-cache headroom, and the roughly 6x throughput gap between them tracks that storage gap rather than their compute gap. Decode here is I/O-bound. Which means we can do the math on paper before buying any hardware.

The per-token byte budget​

Gemma 4 26B-A4B has 30 MoE layers, each with 128 routed experts of which the router activates 8 per token. Each expert is a small MLP (hidden 2816, intermediate width 704) weighing about 11.3MB in FP16 — call it ~3MB at 4-bit. So per generated token, before cache hits:

• Expert traffic: 8 experts × 30 layers × ~3MB ≈ 0.7GB
• Shared-core re-read: attention, shared expert, router ≈ 1.35GB
• Total weights touched per token: ≈ 2GB

An independent estimate from the gemma4-engine project lands in the same place: ~2.5GB per token at Q4 quantization. Now divide by storage bandwidth. The observed 5–6 tok/s on the M2 Air implies roughly 0.3–0.5GB of actual SSD miss traffic per token after the LFU cache and the OS page cache absorb the hot fraction — the cache is doing most of the work, and the miss stream is what sets the ceiling. The M5 Pro's 31–35 tok/s is the same equation with much faster storage and a much bigger page cache on a 24GB machine.

This gives the fleet-sizing formula the rest of the post hangs on:

tokens/sec ≈ SSD sequential-read bandwidth ÷ per-token miss traffic

Everything else — CPU choice, RAM tier, even GPU absence — is second-order. If you know your drive's sustained read rate and you measure (not guess) your runtime's miss traffic, you know your throughput envelope before you serve a single request. That is a nicer position than GPU sizing ever puts you in, where the binding constraint is usually memory capacity you cannot page your way out of.

Prior art: the idea is old, the measurement is new​

Nothing about the core trick is novel, and the post would be dishonest without saying so. EdgeMoE (2023) partitioned MoE models across the storage hierarchy — non-expert weights in memory, experts on flash, fetched on activation — with expert-wise bitwidth adaptation to cut I/O further. MoE-Lightning optimized GPU–CPU–I/O overlap for offloaded inference; Mixtral Offloading made the pattern popular with practitioners; llama.cpp and MLX both support forms of weight offload today. The research frontier has even started grading the idea's downsides: an August 2025 energy-efficiency paper quantified what SSD offloading costs in decode-stage joules per token versus keeping weights in DRAM.

TurboFieldfare's contribution is execution and evidence, not theory: a from-scratch Metal runtime that proves the pattern end to end on consumer hardware, with public benchmarks, a reproducible experiment log, and an OpenAI-compatible loopback server so the thing is actually usable. For a platform team, that distinction matters. Adopting "SSD-resident experts" as a sizing strategy no longer means trusting a paper's simulator — it means porting a pattern with measured numbers attached. The lineage also tells you where the sharp edges are already mapped: prefetch prediction, cache policy, and batching behavior under offload all have prior work to steal from.

Sizing the cheapest fleet node​

Here is the deliverable: what box do you buy if you take this pattern seriously? The spec sheet inverts normal inference shopping. RAM drops to nearly irrelevant — 8GB runs the flagship demo, so any 64GB server has headroom for the OS, the runtime, generous page cache, and concurrent services. CPU matters only modestly; decode of a 4B-active model is light compute. The binding spec is sustained NVMe sequential-read bandwidth, with capacity a distant second (~15GB per model install — trivial).

Concretely, on Hetzner's current dedicated lineup, an AX42-class box (Ryzen 7 PRO, 64GB DDR5, 2×512GB NVMe, roughly €49/month) already clears every requirement with room to spare. Its NVMe array delivers multi-GB/s reads, which the formula above converts directly into a single-digit tok/s envelope — the same neighborhood as the M2 Air result. Compare that against the GPU-first default: a single 24GB cloud GPU typically rents for several hundred dollars a month, an order of magnitude more, to serve a model class this approach runs on commodity flash.

The honest throughput envelope, stated as workload fit rather than hype:

The pattern wins exactly where a self-hosted PaaS has been weakest: offering some local model story without provisioning GPU hardware for it. A background agent tier — sandbox sidecars that reason slowly, cron-like agents, eval harnesses — can live on the NVMe box you already own. Interactive serving still wants VRAM. That split is a feature, not a failure: it lets you stop pricing your cheapest inference tier as if every token were urgent.

Honest limits​

Four caveats, in decreasing order of how much they should slow you down.

Batch-1 decode, no multi-tenant story. Everything measured so far is one sequence at a time. Batching under expert offload is genuinely hard — different sequences route to different experts, which multiplies miss traffic and thrashes exactly the cache the whole scheme depends on. Until someone publishes batching numbers for SSD-resident experts, assume one stream per SSD and size tenant count as one.

The runtime is Apple-Silicon-only. TurboFieldfare is Swift, Metal 4, macOS 26, arm64-only. Your Hetzner fleet runs x86 Linux. The pattern ports — mmap the experts, page them through the OS cache, overlap prefetch with compute — and llama.cpp plus MLX-class offload paths give you starting points, but nobody has published the Linux port with numbers yet. Treat the M2 Air figures as the shape of the curve, not a quote for your hardware.

KV cache eats the budget as context grows. The ~2GB figure assumes a 4K-token KV cache. Long-context agent traces grow that cache linearly, and every gigabyte the KV cache takes is a gigabyte the expert cache loses — which raises miss traffic, which lowers throughput. Sliding-window attention (Gemma 4 uses a 1024-token sliding window with global layers every sixth layer) softens this, but a 128K-context workload on a 2GB budget is fantasy. Size KV headroom from your actual context distribution.

SSD endurance is the dog that doesn't bark. Inference from flash is reads, and reads don't consume program/erase cycles — TBW ratings meter writes. At ~0.5GB of reads per token, even a million tokens a day is half a terabyte of reads daily against drives rated for hundreds of TBW written. The energy paper's decode-joules caveat is real but second-order at this scale. If someone on your team raises SSD wear as the objection, the math says to look at batching and latency instead — those are the actual walls.

What to try Monday morning​

Three steps, in order, each under an hour except the last:

1. Measure your SSD bandwidth first. fio sequential-read on the NVMe volume you already have. That one number, divided by ~0.5–2GB/token, brackets every throughput claim in this post for your hardware. If your drive sustains 3GB/s, you already know the ceiling is single-digit tok/s — no model download required.
2. Replicate the pattern on hardware you own. You don't need a Mac: take an existing NVMe box, load Gemma 4 26B-A4B (or any 128-expert MoE with a 4-bit quant) in llama.cpp with aggressive offload-to-mmap settings, and watch iostat during decode. You are validating the formula, not matching Apple's numbers — if throughput moves with storage bandwidth and not CPU frequency, the pattern holds on your fleet.
3. Pick one batch-shaped workload and cost it. Take your cheapest current GPU-backed job — nightly evals, background summarization, slow agent loops — and price it against an idle NVMe box at ~€49/month equivalent. If the job's latency tolerance clears the tok/s envelope from step 1, you have found your first SSD-streamed tier.

The broader lesson generalizes past this one runtime: sparse models turned memory capacity from a wall into a cache-sizing question, and caches are priced in the cheapest tier of the storage hierarchy you can tolerate. TurboFieldfare just drew the curve with unusual clarity — 26B parameters, 2GB of RAM, and a throughput number you can derive from a storage spec sheet.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Cheap NVMe boxes that serve real models are exactly the kind of hardware it loves to schedule. Star the repo on GitHub or deploy your first app today.