Dora NodaAt 08:39 UTC on July 2, 2026, Railway's network in one US East availability zone lost its route to the internet for about 20 minutes. By the time Railway's own incident report called the incident resolved, storage throughput had spent roughly two hours capped at a third of normal capacity, I/O wait had spiked to 58%, and around 20,000 private-network tunnels were silently blackholed. The routing problem itself was fixed in 20 minutes. The outage it triggered ran six times longer — and the reason why has nothing to do with the ISP that started it.
That gap is the interesting part. A short, boring network blip turned into hours of degraded disks and dead tunnels because of a "hidden bug" — Railway's own phrase — sitting in how Linux hosts pick a network route and then never bother to check it again. It's a mechanism every platform running its own fleet has sitting in its infrastructure too, whether anyone's looked for it or not.
The timeline: a 20-minute fix, a 4.5-hour incident
Here's what Railway's report lays out, compressed to the load-bearing timestamps:
- 07:44 UTC — Packet loss and elevated latency show up on one of the ISPs connecting Railway's datacenters to the internet. Traffic starts rerouting through alternative carriers.
- 08:39 UTC — At the affected US East zone, a secondary carrier gets disconnected during the reroute. The zone is left with no stable default route to the internet.
- 08:59 UTC — The secondary carrier reconnects. Routing stabilizes. This is the 20-minute network problem, and it's over.
- 09:00–10:45 UTC — Storage throughput stays degraded — down to roughly a third of capacity, with I/O wait climbing to 58% — despite the network being healthy again.
- 10:45–11:00 UTC — Railway manually terminates the stuck storage connections still stranded on the wrong path and lets them reconnect cleanly.
- 11:04–11:49 UTC — I/O wait normalizes; engineers identify the parallel private-networking problem and restart mesh networking agents fleet-wide.
- 12:01 UTC — Full recovery.
The "roughly two hours" of customer-visible degradation that Railway's report describes runs from the 08:39 route loss to the ~10:45–11:00 window when stuck connections were forcibly cleared — the disk-performance hit specifically. The private-networking cleanup ran a bit longer, into the 11:49 mesh-agent restart, pushing full resolution to 12:01. Either way, the pattern is the same: a 20-minute network event that was fully fixed by 08:59 kept degrading customer workloads for close to two more hours, and nothing about the network itself was broken during that window. The bug was downstream.
The mechanism: two networks, one bad assumption, and connections that never look back
Railway's datacenters run two physically separate networks per server: a high-bandwidth fabric that carries production traffic, and a much slower management network meant only for administrative access — the out-of-band path engineers use to reach a box when the real network is down.
When the fabric's default route vanished at 08:39, the affected servers' operating systems did what Linux does by default: they fell back to the only route still available, the one on the management network. That part is arguably correct behavior — a route is a route, and staying reachable beats going dark.
The bug is what happened next. Linux defaults to what's called the weak host model: an interface will answer for any IP address configured on the box, not just the one that interface owns. So once traffic was flowing over the management network, the servers happily answered for their storage addresses on that link too — a link with a small fraction of the fabric's bandwidth. Storage I/O that should never have touched the management network started flowing over it anyway, because nothing on the host was configured to refuse it.
That alone would have self-healed at 08:59 when the fabric route came back — except for a second, compounding property of how TCP works: established connections don't re-evaluate their route. A connection opened during the 20-minute window stayed pinned to the management-network path it started on, even after a healthy fabric path existed again, right up until something closed it. Nothing closed it automatically. Railway had to go find and manually terminate the stuck connections at 10:45 before storage performance recovered — a full 106 minutes after the network itself was fine.
Private networking failed for a related but distinct reason: during the routing instability, tunnel traffic briefly passed through a device that rewrites source addresses (a NAT gateway). Roughly 20,000 host-to-host tunnels learned that device's address as their peer's address during the brief window it was in the path, and kept using it — a stale, blackholed address — even after routing rolled back to normal. That's why fixing the network at 08:59 didn't fix private networking either; it took an explicit fleet-wide restart of the mesh networking agents at 11:49 to force every tunnel to re-learn its real peer.
Three failure modes, one root cause each, all invisible from the network layer once you're only watching whether the network itself is "up":
| Failure | Why the network fix at 08:59 didn't help |
|---|---|
| Storage capped at ~⅓ throughput | Weak-host model let storage traffic silently land on the slow management network |
| I/O wait at 58% | Established connections keep their original route until closed — nobody closed them |
| ~20,000 tunnels blackholed | NAT device address got cached as the "real" peer address and never expired |
What a hosted platform's dashboard can't tell you
If you were a Railway customer at 09:15 UTC, your dashboard told you disk was degraded and private networking was flaky. It could not tell you why, because the why lived two layers below anything a tenant can see: which physical network interface a given TCP connection happened to be pinned to, and whether that connection was opened before or after 08:59. That distinction determined whether your workload recovered instantly or stayed broken for another 90 minutes, and it was completely invisible from inside the platform's abstraction.
This isn't a knock on Railway's transparency — the incident report itself is unusually detailed, and publishing "a default Linux behavior let servers silently answer for storage traffic on the wrong network" is the kind of admission a lot of platforms wouldn't make. The point is structural, not about any one vendor: a shared, multi-tenant control plane is a black box by design. You get the platform's summary of what happened, on the platform's timeline, after the platform's engineers have already found and fixed it. You cannot inspect the routing table yourself, you cannot decide to keep your own connections open or force-close them, and you have no lever to pull except reading the postmortem afterward.
Testing the failover path you already have
The uncomfortable generalization is that Railway didn't have a missing backup path — it had one, and it worked exactly as configured: when the primary route died, traffic failed over to the management network rather than going dark. The failure was that nobody had tested what "failover" actually looked like under real storage load, so the interactions between the weak host model, sticky established connections, and NAT address caching never surfaced until 20,000 tunnels hit them simultaneously.
A Cluster API fleet running on owned Hetzner or bare-metal hardware has the identical shape of risk, just with different node names. Concretely, that means testing for the same three failure modes before a real incident finds them:
- Pull the default route on a live node and watch what happens to storage traffic, not just whether the node stays reachable. If your storage/CSI traffic silently reappears on a management or fallback interface, you have Railway's bug. The fix is enforcing the strong host model on interfaces that shouldn't answer for other addresses —
net.ipv4.conf.<if>.rp_filterandarp_filterset explicitly, rather than trusting Linux's default weak-host behavior to sort it out. - Chaos-test connections that are already open when a routing change happens, separately from new connections made after. "The route recovered" is not the same claim as "every open connection recovered" — a game day that only kills and restores a node cleanly will never reproduce this, because it never leaves a batch of established connections mid-flight during the transition.
- Add alerting on your backup/management network's own utilization. If traffic that should be zero-to-negligible on that path starts climbing, that's the leading indicator — it would have flagged Railway's storage traffic silently moving onto the management network before customers ever saw degraded disk latency.
- Include tunnel/overlay-network address learning in the drill, not just node and storage failover. Anything that caches a peer address — a mesh network agent, a CNI overlay, a service mesh sidecar — can pin itself to a transient NAT hop the same way Railway's private networking did, and won't self-heal without an explicit restart.
None of this is exotic. It's the same chaos-engineering discipline — inject the failure, watch what actually breaks, fix what you find — that's been standard practice for a decade. The specific value of Railway publishing this postmortem is that it names the exact place most of those drills stop short: they test whether failover happens, not what failover carries once it's holding production traffic it was never sized or configured for.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Owning the fleet doesn't automatically mean your failover paths are tested; it means you're the one who has to test them. Star the repo on GitHub or deploy your first app today.
Sources:
