Skip to main content

Railway's 8-Hour Outage Was a GCP Account Suspension, Not a Railway Bug

9 min readDora NodaDora Noda
Share

At 22:20 UTC on May 19, 2026, Google Cloud's automated systems suspended Railway's production account without warning. Railway spends roughly $2 million a month on GCP. None of that mattered — an automated action flagged the account, no human reviewed it first, and eight hours later 3 million users had watched their dashboard, API, deployments, and databases go dark.

Here's the detail that should worry anyone running a multi-cloud platform: Railway runs workloads on AWS and its own bare metal too, not just GCP. Multi-cloud was supposed to be the insurance policy. It didn't pay out. The outage cascaded to every provider Railway runs on, because the thing that broke wasn't a region or a data center — it was the control plane, and the control plane was a single tenant inside one hyperscaler's account system.

The Mechanism: Why Multi-Cloud Didn't Save Railway

Railway's edge proxies don't look up "which server should handle this request" from a static list. They pull routing state from a control-plane API — and that API ran on GCP. When Google suspended the account, the API stopped answering. It didn't matter that the workloads behind those routes lived on AWS and on Railway's own bare-metal servers; the map telling traffic how to reach them lived on GCP, and the map was gone.

Here's how it unfolded, second by second, according to Railway's own incident report:

  • 22:10 UTC — Automated monitoring detects API health-check failures; on-call is paged.
  • 22:11 UTC — The dashboard starts returning 503s. Logged-in users can't reach anything.
  • 22:19 UTC — Root cause identified: the GCP production account is suspended.
  • 22:22 UTC — Railway files a P0 support ticket with Google.
  • 23:54 UTC — Disks are restored to a ready state — but that's the easy part.
  • ~01:30 UTC (May 20) — Core networking and edge routing are back. This is the real recovery point, over three hours after the initial failure, because rebuilding routing state took far longer than restoring compute.
  • 07:58 UTC — Incident fully resolved. Total elapsed time: roughly eight hours.

Notice the gap between "disks restored" (23:54) and "routing restored" (~01:30). That 90-minute-plus stretch is the cost of a control plane with a single point of failure: it's not enough to bring the underlying machines back, because nothing can find them until the routing layer that maps requests to machines comes back too. AWS and bare-metal capacity sat healthy and idle for over an hour, unreachable, because the only system that knew where they were had been switched off by someone else's automation.

This is the core lesson, and it's easy to state precisely: multi-cloud protects you against a regional or provider-level infrastructure failure. It does not protect you against a control-plane failure, if that control plane itself is a tenant of one provider. Those are different failure domains, and Railway's architecture only insured against the first one.

It's worth being precise about why route caches exist at all, because the same design shows up in most large platforms and isn't itself the mistake. Edge proxies cache routing state instead of querying the control plane on every request, because a live lookup per request would make the control plane a latency bottleneck and a single point of failure for every request, not just the ones issued during an outage. The cache is the right call for normal operation — fast reads, resilience against brief control-plane hiccups. The failure mode only appears when the source of truth behind the cache is unreachable for longer than the cache's TTL, and there's no fallback path for refreshing it from anywhere else. Railway's proxies did exactly what they were built to do; they kept serving stale-but-correct routes until those routes expired, and then had nothing to replace them with, because the only place to ask was the suspended account.

The Retry Storm That Extended the Outage

Recovery had a second failure mode stacked on top of the first. As routing came back and clients across the internet retried failed logins, builds, and API calls simultaneously, the burst of traffic hit GitHub's rate limits. At 02:47 UTC, GitHub began throttling Railway's OAuth and webhook integrations — the exact systems Railway needed to let users log back in and trigger new builds. Some users who'd been down for over four hours found that even after Railway's own systems recovered, they still couldn't log in or deploy, because a third-party dependency was now rate-limiting the recovery traffic itself.

This is a familiar shape in outage postmortems: the initial failure creates a synchronized retry storm, and the retry storm becomes its own incident. It's worth naming because it means the effective outage window for a meaningful slice of users was longer than the eight hours Railway reports for its own systems — recovery isn't finished when your systems are back, it's finished when every dependency your recovery path touches has absorbed the retry burst too.

A Sharper Failure Mode Than a Regional Outage

2025 and 2026 have had no shortage of major cloud outages to compare this against. AWS's October 2025 US-EAST-1 DNS failure took down Snapchat, Fortnite, Coinbase, and multiple banks. Azure's October 29, 2025 outage, blamed on an "inadvertent configuration change," knocked out Microsoft 365, Xbox, and Starbucks' ordering systems for over ten hours. Both were bad. Both are also, structurally, a different and less dangerous kind of failure than what hit Railway.

Regional and configuration-change outages are bounded by geography or by the blast radius of a specific bad deploy. A well-architected multi-region, multi-cloud system can route around them — that's precisely the failure mode multi-cloud is designed to catch, and in most cases it works.

An account suspension is bounded by account, not by region or provider. Google didn't take down "GCP's us-central1 region" — it took down every resource, in every region, associated with one customer ID, in one action. If that account also happens to host the control plane your other providers depend on for routing, the blast radius isn't "one region of one provider." It's "every provider you run workloads on, because they all take orders from the thing that just got switched off." An account-level kill switch crosses regions and providers by design — that's the entire point of an account boundary — which is exactly what makes it a sharper version of "who can turn off your app" than a regional incident ever is.

What Owning the Machines Actually Removes

The fix isn't "use more regions" or "add a fourth cloud provider as a backup" — Railway already had multiple providers, and it didn't matter. The fix is architectural: the control plane itself has to stop being a tenant inside any single provider's account system.

Railway's own remediation, laid out in its incident report, makes this explicit. The company is demoting GCP to a secondary/failover role and removing it from the data plane's hot path entirely, while re-architecting the split between control plane and data plane so that neither depends on one vendor's account staying in good standing. That's not a bigger multi-cloud footprint — it's a structural change in who can revoke access to the thing that routes your traffic.

If the machines running your control plane are ones you own outright — bare metal you lease or run in a colo, servers under a Cluster API–style declarative lifecycle rather than a hyperscaler's managed control plane — there's no account for a hyperscaler's automation to suspend. A billing dispute, a false-positive abuse flag, a compliance review gone wrong: none of these can touch infrastructure that was never inside that provider's account boundary to begin with. This is the concrete thing "own your machines" buys you that "run on more than one cloud" doesn't: it removes an entire class of failure — the account-level kill switch — that no amount of regional or provider diversity can route around, because that diversity is still expressed as tenancy inside accounts someone else controls.

This distinction matters more, not less, as more of the traffic hitting a platform's control plane comes from automated systems rather than people clicking around a dashboard. An AI agent that deploys and operates infrastructure on your behalf — provisioning a service, rolling a config change, scaling a fleet — is making control-plane calls just as continuously as a human operator, and it inherits the exact same blast radius if that control plane can be switched off by someone else's account-review automation. A platform whose control plane runs on infrastructure it owns gives both human and agent operators the same guarantee: the thing they're talking to doesn't have an off-switch that belongs to a third party.

A Checklist for Auditing Your Own Control Plane

Before assuming your own multi-cloud or multi-region setup would fare better, trace the actual dependency chain:

  • What resolves your routing? If your edge proxies, load balancers, or service mesh pull configuration from an API hosted inside one provider's account, that account is a single point of failure regardless of how many providers host your workloads.
  • What happens to cached state when that API goes dark? Railway's routes had a TTL; when it expired mid-outage, healthy workloads became unreachable. Know your own cache lifetimes and what "worst case, the control plane never comes back" does to them.
  • Which of your dependencies retry synchronously on recovery? OAuth providers, webhook integrations, and third-party auth are exactly the systems likely to get hit by a retry storm the moment your own systems come back — model that as part of your recovery time, not as background noise.
  • Could one provider's admin console, by itself, take down workloads hosted somewhere else? If the answer is yes because of a shared control plane, you have the same structural exposure Railway did — irrespective of how many clouds appear in your architecture diagram.

None of this is a call to abandon multi-cloud. It's a call to be honest about which failure mode it actually insures against — and to notice that "own the machines the control plane runs on" is the only thing that removes the other one.


Bex.co is the open-source, AI-native Render alternative built on this exact principle: push a git repo, get a running HTTPS service on machines you own, with a control plane that isn't a tenant inside any single hyperscaler's account system. Star the repo on GitHub or deploy your first app today.

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex