AWS Fixed the Network in 20 Minutes. Your App Stayed Down for 10 Hours.

AWS marked its own outage resolved at 12:12 UTC on July 24, 2026. The last downstream provider caught in the blast radius didn't mark its incident resolved until 21:28 UTC — 9 hours and 16 minutes later. Between those two timestamps sits everything a status page doesn't tell you.

That day, networking hardware on the path between AWS us-west-2 (Oregon) and the Seattle Metro failed at 10:55 UTC. For most customers the routing blackhole lasted about 20 minutes. AWS was engaged by 11:01, traffic started returning at 11:15, and core routing was back by 11:59 UTC. AWS's public timeline closed the book at 12:12 UTC. If you read only the hyperscaler, the incident was a blip.

If you ran a service on top of that hyperscaler, it was a workday.

IncidentHub tracked 9 confirmed cascade incidents across 7 downstream providers — StatusHub, Cube Cloud, Bambu Lab, SparkPost, FortiSASE, NinjaOne, and SendGrid — plus 3 more possible cascades. Eleven of the twelve were already open before AWS posted its first public update at 11:40 UTC. The shortest upstream outage in that week's trio produced one of the longest downstream tails.

This post maps the gap, explains why a network fix doesn't end an outage, and what it means for teams recovering on hardware they own.

What Actually Broke: Seattle Path Hardware, Not the Cloud​

The failure wasn't a region-wide control-plane collapse or a bad deploy. It was narrower and more physical than that: networking hardware on the path between us-west-2 and the Seattle Metro — specifically through Equinix EqSe2 and the Westin Building Exchange — went down.

That specificity matters for two reasons.

First, the blast radius was geographic, not logical. Traffic inside the region was largely unaffected. What broke was connectivity between the region and the Seattle Metro PoP where Direct Connect and edge interconnections converge. Workloads talking inside us-west-2 could look healthy while anything transiting that fabric — including the AWS Management Console for a subset of customers — could not. Some teams couldn't even see the incident clearly while inside it.

Second, the recovery had a split. Most customers saw routing restored after roughly 20 minutes (10:55 to about 11:15 when traffic began returning, fully back by 11:59 UTC). A smaller population on AWS Direct Connect through the affected Seattle path stayed impacted for 1 hour and 17 minutes. AWS's own accounting later listed 10 affected offerings tied to the event. IncidentHub's separate count — 9 cascade incidents across 7 providers — isn't the same number because it's measuring something different: not what broke inside AWS, but how many other providers had to open their own incidents because of it. Conflating those two counts is the first mistake in reading the incident as "small."

The narrowness is the point. A single piece of networking hardware on a single metro path can be small inside one provider's infrastructure map and enormous downstream, because that path is shared fate for every tenant that transited it.

The Timeline That Matters: Provider Green vs. Tenant Green​

Here's the core artifact — AWS milestones against the downstream incidents that were still red long after AWS went green.

The cascade layer, as tracked by IncidentHub and corroborated by provider status pages:

• 9 confirmed cascade incidents across 7 providers: StatusHub, Cube Cloud, Bambu Lab, SparkPost, FortiSASE, NinjaOne, SendGrid.
• +3 possible cascades under investigation (same time window, same path, weaker causal attestation).
• 11 of 12 tracked downstream incidents opened before 11:40 UTC — before AWS had told the public anything was wrong. If you waited for the hyperscaler status page to tell you to start incident response, you were already late.
• Last to recover: NinjaOne at 21:28 UTC — roughly 10 hours 33 minutes after the initial failure, and 9 hours 16 minutes after AWS's own resolved marker. The single-provider tail is the number your SLO actually felt.

A second framing makes the distortion clearer: the July 24 us-west-2 event was the shortest upstream window among three hyperscaler incidents that week, yet it produced one of the longest downstream tails. Duration of the root cause and duration of tenant pain are not the same metric — and the ratio can invert.

Reading the AWS row alone answers "when did the network come back?" Reading both rows answers "when could a tenant serve traffic again?" Those are different outages.

Why a Network Fix Doesn't End an Outage: Three Tail Mechanisms​

Once routing is restored, you don't get an instant reset to the pre-outage steady state. You get three compounding recovery loads that hit simultaneously, on a system now shared by every tenant that was just starved at once.

1. The reconnection backlog — everyone's thundering herd at once​

During the blackhole, clients didn't stop trying. They queued, timed out, and retried. Connections that would normally arrive spread over minutes piled up as backlog. When the gate reopens, they all arrive at once.

Every tenant's agents, proxies, SDK clients, and browser tabs that failed for 20 minutes now reconnect in the same seconds. For a multi-tenant provider behind the affected path, that's thousands of customers converging on one recovering edge — a thundering herd even if no single tenant is abusive. Health checks pass simultaneously, load balancers re-add targets en masse, and handshakes that normally stagger now co-occur.

The network fix removes one bottleneck but creates another at the next layer up.

2. Queue drains — work didn't disappear, it accumulated​

For asynchronous systems — email (SparkPost, SendGrid), device queues (Bambu Lab), job queues (Cube Cloud), monitoring ingestion (StatusHub, NinjaOne) — the outage didn't discard work. It deferred it.

Messages, webhook deliveries, device heartbeats, and telemetry samples that failed during the 20-minute window now sit in queues. Draining is bounded by downstream capacity: DB write throughput, API rate limits, deduplication, ordering. Drain time scales as outage duration × ingest rate, but drain rate is capped by the same infrastructure that was just impaired.

The October 2025 AWS us-east-1 DynamoDB-rooted outage is the textbook case: AWS fixed DNS in ~2.5 hours, but the region stayed metastable for 15 hours because backlog exceeded processing capacity and retries amplified load. July 24 is smaller but structurally identical — "early signs of recovery" coexisted with hours of elevated errors while backlogs drained.

3. Retry storms — the fix amplifies the load​

Naive retry logic turns a transient failure into a sustained one. When every client retries on failure with similar backoff, retries synchronize and amplify. A system that was capacity-starved during the outage is now hit by more load than during steady state, precisely as it is most vulnerable.

Cisco's analysis phrased it bluntly: initial problems create long-tail impacts that persist after the first fix — shedding retry amplification matters as much as restoring connectivity. Without jitter, exponential backoff, circuit breakers, and bounded concurrency, recovery generates its own outage. AWS's 2017 S3 index-subsystem incident showed the same at a lower layer: GET/LIST/DELETE recovered by 12:26 p.m. Pacific, but PUT lagged to 1:54 p.m. because the placement subsystem couldn't outrun retry pressure.

The July 24 cascade providers hit all three at once: reconnection herds at the edge, queue drains in the async core, and retry storms from every downstream client that had been failing for 20 minutes. None of that work appears on the upstream provider's status page after 12:12. All of it appears on the downstream provider's page for hours.

Reading a Status Page Like an Operator​

There's a 45-minute lesson embedded in the timestamps: AWS impact began at 10:55, first public update landed at 11:40. Eleven of the twelve downstream incidents were already open before that update existed. If your runbook is "watch the hyperscaler status page and decide," you started almost an hour late.

A healthier read of any hyperscaler incident has three rules:

1. Distrust "resolved" as "recovered." Provider "resolved" means the root cause is fixed inside provider infrastructure. It says nothing about whether downstream queues have drained, retry storms have cooled, or error rates have returned to SLO. For the July 24 event, "resolved" overstated tenant recovery by more than nine hours for the tail provider. The October 2025 DynamoDB-rooted outage was worse: 2.5 hours to fix DNS, 15 hours to full recovery. The gap isn't an exception; for any multi-tenant edge, it's the norm.

2. Measure at your layer, not provider layer. If your app is behind an affected path, the only clock that matters is your own: p95 latency to your origin, error rate on your critical path, queue depth on your async jobs, and success rate of the integration call that actually serves your user. Provider green with tenant red is not a contradiction — it's the expected intermediate state while backlogs drain. Teams that paged on their own SLOs during July 24 started remediating earlier than teams that paged on a status-page webhook.

3. Expect convergence pain on shared recovery. When a hyperscaler edge recovers, every tenant recovers through the same edge at once. Your retry budget is not the only retry budget contending for capacity. Rate limits, NAT gateway capacity, auth provider throttles, and webhook delivery concurrency are all shared resources during the tail. Capacity planning for "steady state plus headroom" understates need during recovery by the size of the accumulated backlog divided by acceptable drain time.

If that sounds like it applies beyond AWS — it does. Azure's July 23, 2026 West US incident the day before showed the same shape: IncidentHub detected 19 downstream cascades, with 16 still open after Microsoft's own 19:41 UTC recovery marker. One short incident, many long tails.

The Same Outage on Hardware You Own​

None of this means self-hosting is outage-free. Hardware fails on your side too — single-server self-hosting tools like Coolify, CapRover, and Dokploy wear that reality plainly. The claim isn't "owned hardware doesn't break." It's that when it does, the recovery dynamics are structurally different in three ways that compress the tail.

Your recovery isn't queued behind thousands of strangers. On a hyperscaler edge, your reconnection herd competes with every other tenant's reconnection herd for the same control plane, NAT, and API concurrency. On a single-tenant fleet — a Cluster API (CAPI) fleet on Hetzner dedicated or cloud servers under Cluster API Provider Hetzner (CAPH) — the capacity that was impaired is yours alone to reclaim. When the network path is back, it's your workloads that reconverge, not yours plus ten thousand others. The tail scales with your own backlog, not the platform's aggregate.

Your remediation priority is your own. During the July 24 tail, downstream providers were triaging recovery with no visibility into hyperscaler-internal queue depth and no ability to shed other tenants' load. On owned hardware, priority is explicit: you decide whether to drain critical-path async queues first, shed non-critical retries, scale the node pool, or fail over a location. CAPH expresses that as declarative MachineDeployment and HetznerCluster state — the declared topology is the reconciliation target, not a ticket in a shared support queue. That doesn't make triage free; it makes it local and schedulable.

Your failure domains are correlated on your terms. A 20-minute path failure on a shared metro edge is a single fault domain with thousands of tenants' blast radius coupled by geography they didn't choose. A self-hosted fleet still has failure domains — a switch, a rack, a location — but you chose their boundaries. Multi-location CAPH fallback (e.g., fsn1 ↔ nbg1 ↔ hel1) and headroom kept in reserve rather than scaling exactly to demand are not after-the-fact mitigations; they're the topology you declared before the incident. The cost is explicit on the invoice — a flat monthly rate for a known Hetzner box with 20TB included bandwidth, not a per-GB egress meter that spikes with the retry storm — which is why comparisons against hosted PaaS bills that look competitive in steady state diverge precisely during the tail when traffic multiplies.

That's still a trade. Self-hosting trades opaque shared capacity risk for visible single-provider inventory risk — Hetzner's repeated 2026 capacity notices ("limited availability" on specific cloud server lines) are the owned-hardware version of the same underlying hardware supply shock — and trades managed queue-draining for operational ownership of your own queue-draining. The difference the July 24 timeline makes visible is where the tail lives: stacked behind thousands of other tenants on a recovering multi-tenant edge, or bounded to your own backlog on hardware you can reason about.

Budget for the Tail, Not the Blip​

The July 24, 2026 us-west-2 outage will be remembered as a 20-minute incident because that's what the root-cause duration was. It should be budgeted as a 10-hour incident because that's what the last tenant's recovery took.

If you take one checklist from the gap:

• Retry hygiene is recovery hygiene. Exponential backoff with jitter, bounded concurrency, and circuit breakers aren't steady-state niceties — they determine whether the gate reopening triggers a second outage. The same october 2025 DynamoDB cascade that kept AWS impaired for 15 hours kept impairing downstream tenants via synchronized retry storms long after DNS was green.
• Size queues for outage plus drain, not just steady state. A queue that holds one hour of ingest at steady state is a queue that overflows on a 20-minute outage once retries inflate the effective ingress rate. Keep drain-rate headroom visible — queue depth and age as live metrics, not log entries.
• Run your own red/green signal. Provider status is an input, not a gate. Keep origin-level SLOs (success rate on the call that serves the user) as the paging signal. That buys you the 45 minutes between 10:55 and 11:40 that waiting on a public update costs.
• Choose failure domains you can declare. If your platform can't express "this node pool in this location with this headroom" as checked-in code, you can't change it without a human in the loop during the tail. On a CAPI fleet, that expression is the API — and the invoice is the flat hardware line you already approved, not a new meter you discover in the tail.

The next provider status page will go green quickly. Most do — fixing a single hardware path is fast, and hyperscalers are good at it. The question that determines your outage length isn't how fast the path comes back. It's how long the convergence after it costs you.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Cluster API manages the fleet, Hetzner prices the box, and your recovery runs on hardware you drew the boundary around, not a shared edge you queue behind. Star the repo on GitHub or deploy your first app today.