Skip to main content

Railway's December 16 Cryptominer Incident: How Under 10% Infected Workloads Degraded 100% of Regions

9 min readDora NodaDora Noda
Share
On this page

At 07:05 UTC on December 16, 2025, Railway's monitoring caught something odd: CPU usage spiking across every deployment, in every region, all at once. Five minutes later it was a declared incident. Thirty-two minutes after that, it was a Major Outage with builds paused fleet-wide. The eventual root cause — a cryptominer, smuggled in through a critical Next.js vulnerability — had infected fewer than 10% of Railway's total workloads. Yet the degradation Railway logged was fleet-wide, across all regions, hitting tenants who had never run a line of vulnerable code. That gap — a single-digit infection rate producing a 100%-region incident — is the actual story, and it's a resource-isolation failure mode every shared-compute PaaS has to answer for, regardless of how good its own control-plane code is.

The Timeline

Railway published a postmortem with UTC timestamps for the full incident. Laid out in order, it shows a roughly 4-hour-15-minute arc from first detection to resolution:

Time (UTC)Event
07:05Performance degradation detected; investigation begins
07:10Incident declared: "Degraded Performance across all deployments in all regions"
07:37Escalated to Major Outage; builds paused fleet-wide
07:40Root cause identified: abnormally elevated CPU usage across workloads
08:06Builds re-enabled after initial mitigations
08:30Malicious process identified inside customer workloads (confirmed via eBPF process hooks and sandboxed binary analysis)
09:08Secondary impact concentrated in Europe West (Amsterdam)
11:00Fleet-wide recovery observed
11:20Incident resolved

The declared impact: fewer than 10% of total workloads saw degraded performance, and under 1% of private-network traffic was affected — but that 1% was enough to slow down or drop requests for tenants whose own applications were never touched by the exploit. Amsterdam took the worst of it. The whole thing traced back to a single root cause that had nothing to do with Railway's own control-plane code: a handful of customer-owned Next.js apps still running an unpatched version thirteen days after the vulnerability was public.

The Vulnerability: CVE-2025-55182, "React2Shell"

The CVE at the root of this is CVE-2025-55182 — nicknamed React2Shell — a critical, unauthenticated remote code execution flaw in React Server Components' "Flight" wire protocol, disclosed December 3, 2025. It's an insecure-deserialization bug: a server running RSC fails to validate the structure of a crafted payload before deserializing it, and attacker-controlled data ends up driving server-side execution under Node.js. No login required, no user interaction — one crafted HTTP POST is enough, and independent testing found it worked with near-100% reliability, including against a completely default create-next-app scaffold with zero custom code.

It's a wide blast radius by design: react-server-dom 19.0.x/19.1.x/19.2.x, Next.js App Router from 14.3.0-canary.77 through the 15.x and 16.x lines, plus Vite RSC, Parcel RSC, React Router's RSC preview, RedwoodSDK, and Waku all shipped the vulnerable component. Wiz's scan of live cloud environments found 44% publicly exposed a Next.js instance at all, and 39% of those environments had at least one vulnerable version still running.

Exploitation in the wild started fast — GreyNoise tracked opportunistic, automated attempts from 95 distinct IPs beginning December 5, two days after disclosure, and AWS threat intel flagged multiple China-nexus groups testing early exploit code in the same window. Security researchers eventually catalogued at least six distinct XMRig cryptomining campaigns riding this one CVE, alongside credential-harvesting attempts (environment variables, filesystem secrets, cloud metadata endpoints) and at least one attempted Sliver C2 deployment. Railway itself had pushed alerts to affected customers starting December 3, the same day the CVE went public. By December 16, the attackers who'd been probing for eleven days finally landed a payload against Railway-hosted apps that hadn't patched.

Why Under 10% Infected Became a Fleet-Wide Incident

This is the part worth sitting with, because it's not really a story about a missed patch — plenty of teams miss patches, and the consequence is usually contained to the app that missed it. The consequence here wasn't contained. Railway's own postmortem states the mechanism plainly: "simultaneous execution of these attacks across many workloads and hosts caused fleet-wide resource starvation, which in turn degraded Private Networking in our multi-tenant networking environment."

Unpack what that sentence means mechanically. A cryptominer like XMRig has no incentive to behave like a considerate neighbor — it's stolen compute, and the attacker's payoff scales with however much CPU it can grab. Left unconstrained, it drives every available thread as close to 100% utilization as the runtime will allow. That's fine, in isolation, if the host enforces a hard ceiling on what any one workload can take: modern Linux cgroup v2 exposes cpu.max as a kernel-enforced clamp, not a scheduling hint — a container capped at, say, one core physically cannot burst past it no matter what code is running inside it, malicious or not.

The failure mode Railway describes is what happens when that ceiling either isn't uniformly hard-enforced per tenant, or when tenant compute and shared platform infrastructure share the same physical substrate closely enough that a large-enough simultaneous burst can starve something both tenants depend on — in this case, the private networking layer itself. Kubernetes multi-tenancy research backs up why this is a common trap, not a Railway-specific oversight: namespace-level ResourceQuotas cap the aggregate CPU/memory a group of workloads can request, but they don't guarantee isolation at the node level, and a namespace by default provides no built-in CPU isolation between tenants sharing the same nodes. A burst across "many workloads and hosts" simultaneously — Railway's own phrase — is exactly the scenario where soft, aggregate-level caps aren't enough, because the thing getting starved (private networking) isn't scoped to any one tenant's quota at all.

That's the mechanism behind the headline number: fewer than 10% of workloads were compromised, but 100% of regions degraded, because the failure crossed from "tenant compute problem" into "shared infrastructure problem" — a boundary that per-tenant resource caps and physical workload separation exist specifically to hold.

What Getting Isolation Right Actually Requires

Railway's remediation, per its own postmortem, was reactive and appropriate to the moment: WAF rules to block the known payload signatures, active termination of the malicious binary, automated removal of infected workloads, blocking new builds on still-vulnerable Next.js versions, and heuristic scanners to catch the next variant. Railway also committed to "eliminating the entire class of issues contributing to this incident" — but the published postmortem stops short of naming a specific architectural change to CPU isolation or multi-tenant network separation. That's a reasonable thing to leave out of a public postmortem; it's also exactly the gap worth naming for anyone evaluating a shared-compute platform's resource-isolation guarantees, because those guarantees are what determine whether the next unpatched dependency on someone else's app becomes your incident too.

Concretely, closing that gap means three separable things, and it's worth being precise about which layer each one operates at:

  • A hard per-workload CPU ceiling, enforced by the kernel, not the scheduler. cpu.max under cgroup v2 is a real clamp — a workload configured for one core cannot physically consume two, regardless of what's running inside the container. This is the baseline every multi-tenant compute platform needs regardless of anything else on this list.
  • Physical separation between tenant compute and the platform's own control-plane and networking-plane processes. If the routing layer that every tenant's traffic passes through shares cores with tenant workloads, a big enough simultaneous burst can starve it no matter how tightly each individual tenant is capped — Railway's own diagnosis names exactly this dependency ("degraded Private Networking in our multi-tenant networking environment").
  • Node-pool-level segmentation between tenants or tenant tiers, so the blast radius of a burst — however it originates — stops at a pool boundary instead of reaching the whole fleet. A miner that pegs every core in its own pool is a bad day for that pool's tenants; it should never be a bad day for every region at once.

None of this replaces patching. A compromised app is still compromised, and isolation doesn't un-exploit an RCE — it only bounds what the exploit can cost everyone else. But that's precisely the guarantee worth demanding: your neighbor's missed patch should be their incident, not yours.

The Blast-Radius Question a Dedicated Node Pool Answers Differently

This is the specific place where owning the machines your platform runs on changes the calculus. A Cluster API–managed fleet on Hetzner (CAPH) provisions distinct node pools as ordinary MachineDeployment resources — one tenant tier's workloads land on machines that a different tenant tier's workloads, and the platform's own ingress and control-plane pods, never touch. Kubernetes' own scheduler enforces per-pod CPU requests and limits by default, backed by cgroup v2 on any reasonably current node image, so a compromised app maxing out a miner hits a ceiling Kubernetes itself enforces on that one pod — and at worst saturates the handful of cores in its own node pool, not the shared substrate everyone else's traffic depends on.

That's the concrete difference between "isolation as a policy" and "isolation as a physical boundary": a burst that can't reach another tenant's cores, or the routing layer's cores, because they were never scheduled onto the same machine in the first place. This is not a hosted-versus-self-hosted purity argument — a well-run hosted platform can and should build exactly this separation, and nothing here claims Railway's architecture is worse than any specific alternative's. It's that a team provisioning its own Cluster API–managed fleet decides where that boundary sits, instead of trusting a shared multi-tenant networking layer it can't see the internals of.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, provisioned through Cluster API onto dedicated node pools you control. Star the repo on GitHub or deploy your first app today.

Sources

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex