A Mini-PC With 128GB of Unified Memory Serves MoE Models at 63-97 Tok/s: What Strix Halo Does to Self-Hosted Inference Economics

A mini-PC that fits under a monitor now serves a 30-billion-parameter mixture-of-experts model at 97 tokens per second and a 35B MoE at 63 — measured, reproduced, and published with raw logs. The machine is any 128GB Strix Halo box (Framework Desktop, GMKtec EVO-X2, Beelink GTR9 Pro): a Ryzen AI Max+ 395 whose 128GB of shared LPDDR5X unified memory, carved with a UMA frame buffer, is the reason 120B-class weights fit on one compact PC at all. The 2026 guide wave converging on this claim is unusually evidence-dense, and this post reads it the way a platform team should: what the numbers prove, what the box costs against the meter, and where the story still breaks.

Here is the verdict first. The box is now fast enough to be a credible always-on inference node — that is what actually changed. But on pure token arbitrage at 30B-class hosted prices, the box loses: a $3,000 machine needs years to pay back against a $0.25-per-million-token meter even at high utilization. It wins on privacy, fixed cost, and freedom from rate limits — and the math turns competitive in the 120B reasoning class, where hosted meters run 3-6x hotter. The break-even table below shows the work; the rest of the post supplies the numbers behind it.

Two notes keep this honest. First, throughput is the binding constraint, not just price: a 10-developer team at 2M tokens each per day needs ~230 sustained tokens per second, which exceeds one box's measured aggregate — the team row above is sized to three developers because that is what the hardware fits. Second, add power and your time to the box column: an always-on machine averaging ~120W draws about 90 kWh a month, roughly $15-25 at typical rates, before the afternoon you will spend on BIOS settings (more on that below).

What the guide wave actually proved​

The centerpiece is the strix-halo-guide benchmark campaign on a 128GB Beelink GTR9 Pro, and its headline rows are direct-path numbers — llama.cpp generation speed, not server throughput:

Keep the server paths separate from these direct rows or the numbers stop meaning anything. Through the Ollama API the same Qwen3.6 does ~50 t/s; with 8 parallel clients the box aggregates 173 t/s; an experimental speculative-decoding server route averages 87.5 t/s with a 100.7 best prompt. Each is a different measurement, and the guide is careful to label them — follow that discipline when comparing your own runs.

What elevates this above benchmark theater is independent reproduction. Three matched Corsair AI Workstation 300 systems reproduced the Qwen3-Coder path at 93.6-95.5 t/s, and a GMKtec EVO-X2 on native Ubuntu landed within 2% of the Beelink rows — the guide treats ~2% spread as normal between well-matched systems.

Backend choice matters more than vendor: Vulkan/RADV wins measured generation at 1-4 parallel requests, while ROCm/HIP wins prompt-processing-heavy work and aggregate throughput at 8-16 parallel. If you take one tuning lesson from the wave, it is that one: benchmark your own concurrency shape before declaring a backend winner. And keep the guide's quant discipline: the speed-first Q4_0 rows are faster, but the guide makes no quality claim for them — the balanced UD quants are the default it actually recommends for agent loops.

The stack that makes it infrastructure-shaped​

Speed alone is a demo; the serving stack is what makes the box operable. The easy path is Ollama with the Vulkan backend plus Open WebUI for chat. The serious path is llama-server behind an OpenAI-compatible endpoint — and llama-swap, a zero-dependency Go proxy, hot-swaps multiple models behind one OpenAI- and Anthropic-compatible API so editors and agent loops switch models without restarting anything. That single-endpoint shape is what lets a desk-side box impersonate a hosted API convincingly: point an agent framework at the local base URL and the model behind it becomes a deployment detail rather than a vendor decision.

For models that overflow one box, kyuz0's vLLM toolboxes (511 stars and climbing) package ROCm containers plus a two-node RDMA clustering guide over RoCE, aimed at the trillion-parameter target in AMD's own writeup on running a one-trillion-parameter model locally. Read the measured caveat alongside the ambition: on models that fit on one box, two-node RPC loses 14-22% of single-stream speed, and three nodes are slower still. Clustering is a capacity tool for the 140GB-overflow case the community tested — one box failed, two-node ROCm worked — not a speedup tool. And vLLM itself on Strix Halo remains experimental: the guide's vLLM page is smoke tests plus a bugwatch file, with no throughput claims yet.

Where it still breaks​

Now the price of admission, because there is one and the setup guides are frank about it. The Framework_ROCm and llama.cpp setup guides walk through BIOS (UMA frame buffer to 512MB, IOMMU off), kernel parameters for GTT sizing, and a ROCm install that one community builder summarizes as "the broken ROCm/HIP stack" — which is why the fastest path runs on Vulkan instead, and why ready-made community llama.cpp builds exist: stock releases ship no binaries tuned for Strix Halo. The canonical bring-up looks like this:

# GRUB kernel parameters for large-memory inference on gfx1151
amd_iommu=off amdgpu.gttsize=131072 ttm.pages_limit=31457280
 
# Ollama on the Vulkan backend, HIP out of the way
OLLAMA_VULKAN=1 HIP_VISIBLE_DEVICES=-1 ollama run qwen3.6:35b-a3b
# expect ~50 t/s generation through the API

Then the structural gaps. There is no Kubernetes scheduling story: no DRA-grade device driver, no MIG-style partitioning, no node-pool abstraction — one box is one box, and multi-box means hand-rolled RPC, not a scheduler. Long context costs real speed (32 t/s decode after a filled 128K KV cache on Qwen3.6). And the price line is moving against buyers: 128GB configurations launched near $1,999 and have drifted to $3,299-3,649 for the EVO-X2, $3,449 for the Framework Desktop, and $4,349 for the Beelink GTR9 Pro. Budget the drift, not the launch slide.

Who should buy the box​

With the numbers on the table, the decision matrix writes itself:

• Solo developer running a coding agent. Stay hosted. ~$15 a month of 30B-class tokens cannot justify a $3,000 box on any horizon that matters, and the hosted frontier models are better anyway.
• Team whose code cannot leave the building. Buy the box, and stop doing payback math. Zero-retention enterprise tiers and air-gap requirements are policy constraints, not optimization problems — the news is that the compliant option is now fast enough to use.
• 120B-reasoning power user or sustained background agents. Maybe, at genuinely high utilization. The meter is hot enough (~$1.10/MTok blended) that a box running near its 55 t/s ceiling for months enters payback territory inside its useful life — but verify your sustained tokens-per-second first, because burst-only use never gets there.
• Tinkerer chasing the trillion-parameter demo. Know what you are buying: two boxes, an RDMA setup, and a capacity experiment, not a speedup. Fun, educational, not infrastructure.

The deeper point for a platform team is what the guide wave represents, not any single row in it. Twelve months ago "self-hosted inference" meant a datacenter GPU with a power feed and a driver contract. Today it means a mini-PC with a tuned kernel line — fast enough for agent loops, cheap enough to buy outright, documented enough to reproduce within 2%. The meter still wins on pure tokens at 30B, and saying so plainly is what makes the privacy and fixed-cost cases trustworthy. Own the accelerator where the workload demands it; rent where it doesn't; and re-run this table every hardware generation, because the side that wins keeps changing.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.