Dora NodaAt 22:20 UTC on May 19, 2026, Railway went dark. Not one region. Not one service tier. The dashboard returned 503s, the API stopped answering, and — the part that should stop you mid-scroll — workloads running on AWS and on Railway's own bare-metal fleet died too. The platform stayed down for roughly eight hours, until 06:14 UTC the next morning.
The cause was not a bug. It was not a fiber cut. Google Cloud had suspended Railway's production account — the account the entire platform runs on — as part of an automated Terms of Service enforcement action that Google later acknowledged was applied "incorrectly."
Here is the failure tree that incident exposes, and that almost nobody draws on their architecture diagram:
your app → your PaaS's control plane → your PaaS's cloud account → a hyperscaler's automated billing/abuse algorithm
Every link in that chain can take you down. The first two show up in status pages and SLAs. The third and fourth do not appear in any uptime SLA, any multi-AZ diagram, or any disaster-recovery runbook — and on May 19, the fourth link failed. When you rent a platform, you inherit your platform's standing with its landlord. That is the real lesson of Railway's remarkable, and genuinely commendable, six-month run of published postmortems — and this post walks the whole chain: what happened hour by hour, what the other four major incidents reveal, and what self-hosting your PaaS actually removes from that failure tree (plus what it honestly adds to your pager).
Anatomy of an 8-Hour Blackout
Railway's own incident report is unusually candid, and the timeline is worth reading closely because the shape of the recovery is the story.
- 22:10 UTC — Automated monitoring detects API failures.
- 22:19 UTC — Root cause identified: GCP account suspension. Nine minutes from detection to diagnosis — the engineering response was fast.
- 22:29 UTC — Incident declared; GCP account access restored after a P0 ticket and an account-manager escalation.
- 23:54 UTC — All persistent disks restored.
- 01:38 UTC (May 20) — Networking restored.
- 04:00 UTC — API, dashboard, and OAuth confirmed operational.
- 06:14 UTC — Incident moved to monitoring. Total wall-clock impact: roughly eight hours from first failure to the all-clear (22:20 to 06:14 is 7 hours 54 minutes; call it eight).
Notice the gap. The account was un-suspended within nine minutes of the incident being declared. The platform still took almost six more hours to come back. Why? Because a cloud account suspension is not a power cut you recover from by flipping the breaker — it deallocates and detaches resources underneath a running system. Disks had to be restored, networking rebuilt, and deploys were deliberately paused "to prevent overwhelming systems as queued work attempted to execute simultaneously" — a thundering-herd cold start where every queued job on the platform wants to run at once.
And the blast radius question — why did workloads on AWS and Railway Metal die when only the GCP account was suspended? Railway's report answers it directly: "Railway's edge proxies rely on a Google Cloud-hosted control plane API to populate their routing tables." When the control plane vanished, the proxies coasted on cached routing state — until the caches expired. Then every region, every provider, every tier went down together. Multi-cloud data planes do not save you when the control plane lives in one account.
Six Months, Five Major Postmortems
The GCP suspension was not an isolated stumble. Between November 2025 and May 2026, Railway published postmortems for five major incidents — a transparency streak most platforms would never attempt, and one that gives us a rare labeled dataset of how a fast-growing PaaS actually fails.
| Date | Incident | Duration | Failure class |
|---|---|---|---|
| Nov 20, 2025 | GitHub webhook traffic spiked 10x; deploy queue overloaded, workers locked up under memory pressure | 2h 24m (new deploys only) | Self-inflicted growth pain |
| Jan 26–29, 2026 | GitHub OAuth rate limits exhausted — the dataloader minted ~82 tokens/second against a 2,000/hour limit | ~3 days, intermittent | Self-inflicted growth pain |
| Feb 18–21, 2026 | Nine DDoS waves atop a fiber cut, a failing mitigation vendor, an emergency Fastly WAF rollout (~2,700 endpoints hit SSL errors), and an upstream Cloudflare BGP outage | 4 days of intermittent impact | Adversarial + upstream vendor |
| Mar 30, 2026 | CDN config change cached authenticated responses and served them to unauthenticated users on ~0.05% of domains | 52 minutes | Self-inflicted (data exposure) |
| May 19, 2026 | GCP suspended Railway's production account via automated ToS action | ~8 hours, full platform | Transitive: landlord risk |
(There were smaller ones too — an 8-minute API outage in December from a schema migration locking a billion-row table — but these five are the majors.)
Read as a set, the first four are the ordinary physics of running a multi-tenant platform at high growth: queues that melt under 10x spikes, rate limits you did not know you were consuming, attackers who show up the same week your fiber gets cut, a one-line CDN config that briefly leaks other people's pages. Any platform — including one you host yourself — is exposed to versions of these.
The fifth is a different species. Nothing in Railway's code failed on May 19. No attacker showed up. Their architecture worked as designed. The failure was contractual: an algorithm at their vendor decided their account should not exist for a while. You cannot engineer around that with more replicas, because every replica lives inside the thing that got suspended.
The Fourth Link: Renting Your Platform's Platform
Uptime SLAs are written in the vocabulary of component failure: instance dies, AZ floods, region degrades. Multi-AZ and multi-region architectures are answers to that vocabulary. An account-level suspension speaks a different language entirely — it is not a component failing but the container of all your components being administratively revoked. Multi-AZ redundancy inside a suspended account is redundant nothing.
And SLAs do not really cover it in any sense that matters. A suspension traced to an "automated action" taken "incorrectly" might earn service credits — a percentage discount on next month's bill. Railway's tenants lost eight hours of production; a credit memo does not restore a missed batch window or a churned customer. The SLA is a pricing instrument, not a risk transfer.
What makes the Railway incident such a clean case study is that Railway itself drew the conclusion in writing. Their remediation list is an architecture confession: extend the high-availability database shards across AWS and Metal, build "true mesh networking" so the GCP-hosted control plane is out of the hot path, and ensure "core services, especially user facing components, are not dependent on any one vendor or platform." Translated: the platform you rent was, until May, a single-vendor application with a single point of administrative failure — and its vendor's abuse desk was in your dependency chain.
That last phrase deserves emphasis, because it generalizes. When you deploy on a PaaS that runs on a hyperscaler, your uptime is coupled to things you cannot see or audit: whether your PaaS pays its bill on time, whether some other tenant's cryptominer trips a platform-wide abuse heuristic, whether an automated ToS classifier has a bad day. Every tenant inherits their platform's standing with its landlord. You did nothing wrong on May 19. Neither, apparently, did Railway. Everyone went down anyway.
What Self-Hosting Removes From the Failure Tree — and What It Adds
The reflexive conclusion — "so self-host everything" — deserves the same skepticism as the reflexive "never self-host." Here is the honest two-column accounting, using Railway's own six months as the evidence base.
What moving your PaaS to machines you own removes:
- The fourth link. On owned or directly leased hardware (a Hetzner fleet, colo racks), there is no hyperscaler account whose suspension vaporizes your disks and networking in one administrative action. Your contract is for machines, not for a revocable managed account wrapping every resource you have.
- Other tenants' weather. The Nov 20 deploy-queue meltdown froze deployments for everyone because everyone shares a queue. On your own platform, your queue serves you.
- Someone else's config on your traffic. The Mar 30 CDN incident cached and leaked authenticated pages because of a change you did not make, could not review, and could not decline.
- Someone else's rate-limit budget. The Jan 26 outage was Railway exhausting its GitHub OAuth quota — a shared consumable you draw from without knowing its level.
What it adds — and this list is not small:
- You are now the abuse desk and the DDoS pager. Reread the Feb 18–21 postmortem: nine attack waves, a mitigation vendor failing mid-attack, an emergency WAF migration that broke SSL on ~2,700 endpoints. Railway had a security team doing that for four days straight. Self-hosting means that pager is yours — though you can (and should) still put a WAF/CDN you chose in front.
- Landlord risk shrinks; it does not vanish. A dedicated-server provider can terminate accounts too. The blast radius differs — leased physical machines are not deallocated by a ToS bot the way cloud API resources are, and a provider dispute typically comes with human process and notice — but "zero counterparty risk" only exists for hardware you physically own in space you lease on paper.
- The unglamorous ops floor. Disk failures, capacity planning, backup and restore drills, the billion-row migration that locks a table at 15:18 UTC. Railway's smaller postmortems are a preview of your new job description.
The verdict is not "self-hosting eliminates risk." It is that you are trading counterparty risk classes: swapping an opaque, algorithmic, account-scoped failure mode you cannot mitigate for a set of operational failure modes you can staff, automate, and drill for. For some teams the trade is obviously wrong — a two-person startup should absolutely rent the pager. For teams already running Kubernetes, already carrying an on-call rotation, and increasingly for teams whose AI agents carry chunks of that rotation, the trade keeps getting better — because the ops floor is exactly the part that tooling keeps compressing, while the fourth link stays exactly as opaque as it was on May 19.
The De-Renting Trend Is Coming From Inside the House
The most telling detail in this whole saga: Railway's answer to the incident is to become less of a cloud tenant. The Metal buildout, the multi-provider database shards, the mesh networking that evicts GCP from the hot path — the PaaS vendors themselves are de-renting, at the same time their customers are told that owning infrastructure is an anti-pattern. Watch what they do, not what the pricing page implies.
The playbook Railway is executing for itself — control plane you operate, on providers you can swap, with no single account able to take the whole system administratively offline — is the same playbook now available to any team, because the building blocks (Kubernetes, Cluster API, commodity dedicated servers) are open source and rentable by the machine, not by the revocable account.
Six months of Railway postmortems, honestly published, teach a precise lesson: most platform failures are ordinary and survivable, but the one that took the whole platform down for eight hours was the one no amount of Railway engineering could have prevented in-place — because it lived one contractual layer above their architecture. Whatever you deploy on, know how many links your failure tree really has, and who owns the last one.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with the same trade-offs discussed above, distributed differently: the control plane is yours to operate, and no vendor's abuse algorithm sits in your dependency chain. Star the repo on GitHub or deploy your first app today.
