Dora NodaOn May 19, 2026, at 22:20 UTC, Railway's production account stopped existing. Not its servers, not its regions — its account. An automated Google Cloud enforcement action suspended Railway's production GCP project over a flagged terms-of-service violation, with no prior notice, and within minutes the dashboard, API, control plane, and databases were dark.
Then the outage did something stranger: it escaped Google Cloud. Railway's edge proxies resolve routes through a GCP-hosted control-plane API, so as cached routes expired, workloads running on Railway's own bare metal and on AWS started returning 404s too. Containers still running; nothing reachable. At peak impact, all Railway workloads across all regions were unreachable, and the platform didn't fully recover until about 06:14 UTC — roughly eight hours later, across an estimated 3 million users hosting some 10 million services.
That incident earned 546 points and 345 comments on Hacker News, and it deserves every one of them. But the May outage is only the loudest of five. Railway's policy is to publish a post-mortem for every major outage, and since last November the company has published five of them — a public reading list that traces, better than any architecture diagram, what "hosted PaaS reliability" is actually made of.
Here is the whole list, up front, with each report's own attribution:
| Date | Duration / blast radius | What the post-mortem blames | Report |
|---|---|---|---|
| Nov 25, 2025 | ~2.5h; Free/Trial/Hobby deploys paused | GitHub API latency spiked to ~4x p95 during peak hours → Temporal task-queue workers OOM-crashed in cascade | Incident Report |
| Dec 8, 2025 | ~25 min; dashboard, CLI, login, API down | Nullable-column migration on a ~1B-row table stalled on a lock → PgBouncer exhausted DB connections → backend replicas failed health checks | Incident Report |
| Jan 26–29, 2026 | Multi-day intermittent; GitHub logins + deploys failing at peak US hours | Uncached token dataloader minted ~82 GitHub OAuth tokens/sec → blew past GitHub's 2,000/hour limit | Incident Report |
| Feb 11, 2026 | ~4h; ~3% of fleet force-paused, incl. Postgres/MySQL | New anti-abuse ruleset false-positived on legitimate workloads → erroneous SIGTERM; dashboard showed dead workloads as active | Incident Report |
| May 19–20, 2026 | ~8h; platform-wide, all regions unreachable at peak | GCP auto-suspended Railway's production account (no notice); route-cache expiry cascaded the outage to Metal + AWS; recovery surge got Railway rate-limited by GitHub | Incident Report |
Context matters: this is a platform in furious ascent. Railway raised a $100M Series B in January 2026 as a 30-person company serving 2 million developers and processing a trillion requests, explicitly to expand its private data-center footprint — Railway Metal — and go enterprise.
The post-mortems aren't the story of a platform falling apart. They're the story of what even a well-run, well-funded hosted PaaS is made of: upstream accounts, shared control planes, and provider-operated enforcement. Read all five and a pattern snaps into focus.
What each post-mortem actually blames
November 25, 2025: the queue that ate itself. Railway runs deploys, config changes, and limit updates through a Temporal-backed task queue. Around 19:30 UTC, GitHub API calls slowed to nearly 4x their usual p95 — during peak deployment hours, with no sign GitHub itself was unhealthy.
Workers handling GitHub calls consumed more and more resources, hit out-of-memory failures, and crashed; their tasks shifted to surviving workers, which crashed in turn. New workers spun up, were immediately buried by the backlog, and OOM'd too. Railway paused Free, Trial, then Hobby deploys to relieve pressure, bumped worker CPU/memory, and fixed the worker parameters that let them claim more tasks than they could handle. Full recovery by 01:22 UTC. Notably, running workloads never went down — only the ability to change anything.
December 8, 2025: the billion-row migration. Railway's backend hangs off a shared PostgreSQL database fronted by PgBouncer, with multiple replicas kept in rotation by health checks that each need a DB connection. Engineers shipped a migration adding a nullable column to a heavily used table with roughly a billion rows.
A long-running query held locks on that table, stalling the migration; connection attempts piled up, PgBouncer blew past the connection limit, and replicas started failing health checks and dropping out of rotation. Once the long query finished and released its locks, the migration applied instantly and everything recovered — the whole outage lasted about eight minutes of real downtime inside a 23-minute window. Again, running deployments stayed online throughout; only the dashboard, CLI, login, and API went dark.
January 26–29, 2026: death by OAuth token. This one ran for days. Railway's installationTokenById dataloader created fresh GitHub installation tokens on every request batch without caching across requests — around 82 new tokens per second at peak, worsened by recently scaled-up backend replicas.
That smashed through GitHub's OAuth token rate limit of 2,000 per hour, following an hourly pattern where limits exhausted about 30 minutes into each hour. Users couldn't log in via GitHub, couldn't connect repos, and saw "GitHub repo not found" errors on deploys during peak US business hours. The fix cascade (token caching cut calls to ~10%, concurrency reductions, halved commit-status updates, background-refresh tuning) took days to land with GitHub Support's help, including two rounds of temporarily disabling GitHub logins entirely. Railway's own framing is candid: record sign-up growth meant scale events passed straight through to customers.
February 11, 2026: the abuse robot fires on friendlies. After spotting a new abuse pattern, Railway dry-ran a new anti-fraud ruleset, then began a staged fleet-wide rollout of automated enforcement. The targeting criteria were over-broad: legitimate workloads were put into a "forced pause" state and hit with SIGTERM — including customer Postgres and MySQL databases.
Roughly 3% of the fleet went dark to end users, and to add insult, the dashboard kept showing terminated workloads as active while surviving services threw networking errors calling neighbors that no longer existed. Engineers noticed within 14 minutes, reverted within 23, and ran automated recovery fleet-wide; full recovery by 18:31 UTC. The honest summary: a robot with fleet-wide kill authority and a bad heuristic.
May 19, 2026: the account that vanished. The big one. Google Cloud's automated enforcement suspended Railway's production account over activity flagged as a ToS violation — Railway says it had received previous violation notices and logs existed in Cloud Logging, but no notice preceded this suspension. Everything GCP-hosted died immediately: dashboard, API, databases, GCP compute.
The cascade is the instructive part: Railway's edge proxies depend on a GCP-hosted control-plane API to populate routing tables, so as route caches expired, Railway Metal and AWS-burst workloads — containers still healthy — became unreachable with 404s. Recovery had its own second outage: the retry surge from cleared caches got Railway's OAuth and webhook integrations rate-limited by GitHub, blocking logins and builds again, and ToS-acceptance records were reset, forcing every user to re-accept. Railway's post-mortem takes "full responsibility for the architectural decisions that allowed a single upstream provider action to cascade into a platform-wide outage" and commits to removing GCP from the data-plane hot path.
The pattern across all five
Five incidents, five root causes — and three shared dependencies underneath them.
1. Upstream account standing is load-bearing infrastructure. Three of the five outages are, at bottom, "a third party's account/rate-limit machinery said no." GitHub API latency triggered November; GitHub's OAuth rate limit was January; GitHub rate-limiting struck again as May's second-order outage; and GCP's automated suspension was May.
Your redundancy plan covers dead VMs and unavailable zones. It does not cover "your account no longer exists" — yet that is precisely the failure mode that produced the longest outage in the set, from a vendor Railway reportedly pays eight figures a year. When your platform runs inside someone else's account, their fraud automation is your single point of failure, and you cannot see its rules, canary its changes, or appeal to it in less than eight hours.
2. One shared control plane means one shared fate. November's Temporal queue, December's shared Postgres, May's GCP-hosted routing API: every deploy, config change, login, and route resolution traverses platform-wide systems every tenant shares. Railway deserves credit for the bulkheads that held — running workloads survived November and December untouched — but May showed the limit of the bulkhead that matters most.
The compute was multi-cloud (GCP + AWS + own metal); the routing intelligence was single-homed. Multi-cloud workloads with a single-cloud control plane is single-cloud with extra steps, and the route-cache TTL was the countdown timer on the illusion.
3. Provider-operated enforcement can terminate your workloads. February's abuse ruleset and May's GCP suspension are the same shape at different altitudes: automated enforcement with fleet-wide authority and no human in the loop between "flagged" and "killed." Railway's robot SIGTERMed 3% of its own fleet; Google's robot suspended 100% of Railway's GCP project. In both cases the detection-to-destruction path was fully automatic, and in both cases the humans found out from the wreckage. Any hosted platform concentrates this risk twice — its own enforcement plus its providers' — and customers can inspect neither.
The through-line is the one the TODO spec names: hosted-PaaS reliability is bounded by the reliability of the cloud account underneath it. Not the cloud's infrastructure — GCP's VMs were fine on May 19; GitHub's API was arguably fine on November 25 — but the account standing, rate limits, and control-plane tenancy your vendor inherits from their vendors. Your SLA multiplies down through every account boundary you can't see.
What changes when you own the machines
The honest version of this section cuts both ways, so here is the ledger — which of Railway's five failure modes disappear on a self-hosted fleet on owned hardware, and which you inherit as your own on-call burden:
| Failure mode | On owned machines |
|---|---|
| Upstream account suspension (May) | Gone. There is no account above you to suspend — no provider fraud robot can delete your project. Your machines answer to your API keys, not a ToS classifier. |
| Third-party rate limits in the deploy path (Jan, Nov, May-2nd-order) | Shrinks, doesn't vanish. You still call GitHub for code — but logins can be SSO you control, deploys can be git-push-over-SSH or mirrored repos, and no 2,000/hour shared OAuth pool sits between you and a deploy. |
| Shared control-plane database (Dec) | Yours to operate. Postgres migrations on a billion-row table are exactly as dangerous self-hosted — the difference is you can schedule, canary, and roll back on your own cadence instead of discovering them on someone's status page. |
| Fleet-wide enforcement false positive (Feb) | Impossible from a vendor; possible from yourself. Your own automation can still SIGTERM your fleet — but the ruleset is your code, the dry-run is your staging, and the revert is your call in minutes, not a ticket queue. |
| Single-homed routing intelligence (May cascade) | A design choice you get to make. Keep route state on your own control plane with no expiry-into-nothing, and there is no cache TTL counting down to a 404 cascade. |
The trade is real: owning the fleet means you are the on-call for Postgres, for migrations, for abuse tooling. Nobody publishes your post-mortems for you. What you buy is not the absence of incidents — it's the absence of other people's incidents.
No ToS classifier at a company you pay eight figures to can end your week. No shared OAuth pool you can't inspect can fail your deploys at 30 minutes past the hour. Your outages will be your own bugs, on your own machines, fixed on your own timeline. For a team that can staff that, it is strictly more agency than any dashboard offers.
Questions to ask any hosted PaaS
Railway's five reports double as a due-diligence checklist. Before you bet production on any hosted platform, get answers to these — in writing, not vibes:
- Where does the control plane live? If the routing/API layer is single-homed in one provider's account, multi-cloud workloads are marketing. Ask what happens when that account, not that cloud, has a bad day.
- What happens when caches expire? May's cascade had a TTL fuse. Ask which cached states degrade into "unreachable" versus "stale but serving," and how long each fuse is.
- Can enforcement pause my workloads? February's robot and May's suspension both killed serving automatically. Ask what automated systems — the vendor's and their providers' — hold termination authority over your apps, and whether a human confirms first.
- Which third-party rate limits sit in my deploy path? Three of five Railway incidents touched GitHub's API. Ask for the full list of external APIs in the login/deploy loop and what happens at each limit.
- Do running workloads survive a control-plane outage? November and December passed this test; May failed it. "The dashboard is down but apps stay up" versus "everything 404s" is the single most important bulkhead question you can ask.
Railway deserves the last word of credit: five public post-mortems in six months, each naming its own architectural decisions, is what accountability looks like. Most vendors would have shipped five status-page paragraphs. But transparency about the failure doesn't change the structure that produced it — and the structure, across all five reports, is the same: your apps, running on their account, routed by their control plane, policed by their robots. Read the list once and it's a rough half-year. Read it as architecture review and it's an argument for owning the machines.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
