Skip to main content

Not Us, But Also Not Not Us: Fly.io's July 22 DFW Outage and the Upstream You'll Never Meet

10 min readDora NodaDora Noda
Share
On this page

On July 22, 2026, at around 08:00 UTC, apps running in Fly.io's Dallas region started going dark. Not all of Fly.io — just DFW. By 08:48 UTC the affected machines had recovered, and by 10:05 UTC the incident was marked resolved. Total damage window: roughly two hours on the status page, about thirty minutes of hard connectivity loss in the middle. The cause, per the incident record: emergency maintenance by an upstream network provider — a company no Fly.io tenant ever signed a contract with, picked from a vendor list, or could have routed around.

That sentence is the whole post. Your app was down for half an hour because a company you never heard of did maintenance you never approved, on infrastructure you never chose, in a relationship that exists entirely below the floor of the platform you actually pay. And there was nothing you could click to fix it.

This is a post about inherited dependency: how many hops of "not us" sit underneath a hosted platform's status page, what those hops cost you when one of them moves, and what changes — and what honestly doesn't — when the network provider is the hosting provider.


The two hours, in one ledger

Here is what the public record shows for July 22, 2026 in DFW. The entries come from Fly.io's status history via its mirrors and from Upstash, whose managed Redis runs on Fly.io infrastructure and caught the same blast:

Time (UTC)What the record says
~08:00Connectivity for apps in DFW drops. An upstream network provider begins emergency maintenance affecting the region.
08:03Status trackers log the start of a DFW maintenance event.
08:48Affected Fly.io-hosted machines recover; Upstash reports service availability restored for its Fly-hosted DFW databases.
10:05Incident marked resolved, monitoring ends.

Two numbers matter here. The maintenance window ran about two hours (08:00–10:00 UTC). The hard outage — machines unreachable, Redis timing out, deploys failing — ran roughly thirty to fifty minutes inside it. The gap between those two numbers is itself the story: from the tenant side there was no meaningful difference between "upstream maintenance in progress" and "my app is down." The status page drew a distinction the users couldn't feel.

Note also who else went down: Upstash Redis databases hosted on Fly.io in DFW. That's a second-order casualty — a tenant of a tenant. Upstash customers didn't choose Fly.io's DFW region; they chose Upstash, which chose Fly.io, which depends on an upstream transit provider. Three hops of inherited dependency, and the person getting paged at the end of the chain picked exactly one of them.

The hop map: where "not us" actually lives

To see why nobody could route around this, it helps to draw the chain a request walks to reach a Fly.io app in DFW:

  1. Your app — a Machine (a lightweight VM) on a physical host in a Dallas facility.
  2. fly-proxy — Fly.io's Rust-based edge router, which picks up connections on a global Anycast IP via BGP announcements and forwards them to the nearest healthy Machine.
  3. Regional transit — the BGP peering and upstream providers that carry traffic into and out of the DFW region.
  4. Physical plant — fiber, cross-connects, and the maintenance windows of whoever owns them.

The July 22 failure lived at hop 3. Emergency maintenance by an upstream provider cut the region's connectivity for roughly half an hour. Nothing was wrong with hop 1 (your code was fine), hop 2 (the proxy logic was fine), or hop 4 in the abstract (nobody cut a cable; someone scheduled work and the work bit). Fly.io's own control plane didn't fail. Its vendor's network did — a vendor relationship that exists one full layer below anything a tenant can see, audit, or fail over from.

This is what "the platform's outage was actually its upstream's upstream" means in practice: the status page can only narrate the layers it operates. Below a certain depth, the postmortem necessarily reads "not us, but also not not us" — the failure is inside the platform's blast radius but outside its control plane, caused by a party the tenant has no standing with. You can't open a ticket with your provider's provider.

What a tenant could actually do during the window

This is the section that converts sympathy into engineering. During those thirty-odd minutes, a typical single-region DFW app had exactly one option: wait. Here's why each of the obvious escapes was closed:

  • Fail over to another region? Fly.io Machines are cheap to run in two regions — but Fly.io Volumes are pinned to a single host in a single region. Your stateless web tier can exist in Chicago; your data can't follow it without a second volume, a snapshot restore, or an external replica you built ahead of time. The community's own postmortems from other regional incidents (stuck volumes in EZE, machines wedged in created state in DFW) keep rediscovering the same constraint: the volume lives in one region, and surviving a region outage means provisioning the escape hatch before the outage, not during it.
  • Let Anycast route around it? Anycast steers clients to the nearest announced region. When DFW's transit is degraded rather than cleanly withdrawn, traffic can keep arriving at a region that can't serve it. Graceful degradation of inter-region routing is a hard distributed-systems problem, not a toggle.
  • Redeploy elsewhere? During the window, deploys touching DFW failed — machines stuck, timeouts on start. The escape vehicle was parked inside the burning garage.
  • Wait for the database? If your Redis was Upstash-on-Fly in DFW, your cache/session layer was down on someone else's status page, with its own recovery timeline (restored 08:48 UTC) that you could watch but not accelerate.

The honest multi-region setup — volumes in two regions, snapshots on a schedule (fly volumes snapshots create), a documented primary_region flip runbook — absolutely survives this class of outage. But price it honestly: it doubles your volume spend, it requires rehearsal (an untested failover runbook is a comment, not a plan), and cross-region Postgres failover is something even Fly.io's own docs describe as risky rather than automatic. Most small teams don't carry that standing cost, which is rational right up until the morning it isn't. The outage didn't punish bad engineering. It punished the standard, documented, default architecture.

Counting hops: the transparency ceiling of any status page

There is a general principle underneath this specific Tuesday. Every hosted platform inherits dependencies its tenants never picked — transit providers, DNS operators, certificate authorities, host-level kernel bugs, the building's power feed — and its status page has a transparency ceiling: it can only see, and only disclose, the layers it instruments. Below that ceiling, incidents arrive pre-anonymized as "upstream network provider" with no name, no ticket number you can follow, and no ETA firmer than "monitoring."

Count the hops for a typical small app on a hosted PaaS and the number is sobering: your code → the platform's scheduler → the region's hosts → the region's transit → the fiber owner → the power grid. Six layers, of which you chose one (your code) and contracted for one (the platform). The other four are inherited — and any one of them can page you at 08:00 UTC on a Tuesday without ever having met you.

This isn't an argument that hosted platforms are negligent. It's an argument about legibility: when something breaks at hop 4, the tenant's debugging surface (logs, metrics, fly status) shows symptoms with the cause redacted. You get to watch the "Maintenance" event tick from 08:03 to 10:05 and infer the rest. The cost of inherited dependency isn't just the downtime — it's the downtime plus the forty-five minutes your team spends proving the problem is below your floor before they can stop debugging their own deploy.

What changes when the network provider is the hosting provider

Now the contrast this blog exists to draw. A self-hosted fleet on owned hardware — Cluster-API-provisioned machines on Hetzner, the shape Bex.co targets — collapses one specific hop: the company that operates the network is the company you pay for the server. Hetzner owns its data centers in Germany and Finland, runs its own backbone and peering (DE-CIX, AMS-IX and peers), and sells you the machine and the pipe as one contract. When something breaks, there is one status page, one support queue, and one vendor whose name you already know. The postmortem can't read "our upstream's upstream" because there is no upstream's upstream — there is just the provider, and you.

That is a real, structural difference, and it's worth one fewer layer of anonymized maintenance windows. But honesty requires the other half: it is one fewer hop, not zero hops. Owned hardware still sits on fiber someone can cut, power grids someone can attack, and DRAM markets that can price-spike your next node pool. Hetzner's US and Singapore presence is colocation rather than owned buildings, which reintroduces a landlord. The claim is not immunity — anyone selling immunity from physics is selling something else. The claim is legibility and standing: fewer anonymous layers, a direct contract at every remaining one, and logs, metrics, and machine lifecycle (Cluster API Machines, not opaque provider hosts) that you can inspect instead of infer.

Put as a rule of thumb: count the hops between your deploy and the physical plant, then count how many of those hops you have a contract with. A hosted PaaS gives you one contract covering five hops. Owned hardware gives you fewer hops and a contract at each. Neither survives a backhoe. Only one lets you watch the backhoe on your own monitoring.

The runbook this outage actually justifies

If you run on Fly.io — or Render, Railway, or any region-pinned hosted platform — July 22 justifies four concrete, unglamorous items, none of which require migrating anywhere:

  1. Snapshot schedule with a tested restore. A snapshot you've never restored is a backup-themed wish. Time the restore; write down the number.
  2. A second volume in a second region for anything stateful you can't afford to lose for an hour — or an explicit, written acceptance that one region means one fate. "Single region" is a fine choice when it's a choice.
  3. Status-page literacy in your on-call guide: maintenance events in your provider's region are outage warnings for single-region apps. Treat "Maintenance 2h" the way you'd treat a storm warning, not a changelog.
  4. A blast-radius line in your architecture notes: which of your dependencies are tenants-of-tenants (Upstash-on-Fly, and equivalents)? Each one adds a hop you didn't pick. Know your count.

And if you're the kind of team that reads a thirty-minute upstream maintenance window and thinks "I'd rather own the whole stack down to the machine" — that's a legitimate response, provided it's aimed at the right prize: not fewer outages, but outages with your name on the contract and your own telemetry on the cause.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex