Dora NodaOn July 17, 2026, a subset of new Vercel deployments under Hobby-plan teams started failing function invocations at 10:18 UTC — and the first public status update appeared at 14:12 UTC, nearly four hours later. Six days later, on July 23, telemetry forwarded through Vercel Drains silently stopped delivering for six minutes; tenants learned about it from a retroactive status entry posted the next afternoon, roughly 22 hours after the gap. In both cases the platform-level view said nothing was wrong while a specific slice of tenants quietly lost either compute or observability.
That shared shape is the story: tier-scoped and feature-scoped failures that a green status page cannot catch, because the blast radius is a slice (one plan tier, one forwarding pipeline) rather than the platform. This post reconstructs both incidents from Vercel's own status history, shows the pattern repeating twice more in the same quarter, and ends with four concrete guarantees any multi-tenant deploy platform should build so a scoped failure surfaces to the affected tenant instead of happening invisibly.
The four-hour Hobby window nobody posted about until it was over
The July 17 incident, officially titled "Increased invocation failures for Hobby Team functions", is a textbook tier-scoped failure. Function invocations failed — but only for a subset of new deployments created under Hobby-plan teams. Existing deployments were unaffected, and every paid tier was unaffected. If you checked "is Vercel down?" from anywhere outside that slice, the answer was no.
The timestamps, all UTC on July 17, tell the detection story:
| Time (UTC) | Event |
|---|---|
| 10:18 | First impacted Hobby-team deployments created (window opens) |
| 14:10 | Last impacted deployments created (window closes) |
| 14:12 | First status update: "elevated rates of invocation failures for Hobby Team functions in new deployments" |
| 14:19 | Source identified, fix being implemented |
| 14:27 | Fix deployed, monitoring; new Hobby deployments no longer fail |
| 15:19 | Incident resolved |
Two things stand out. First, the ~4-hour gap between first impact (10:18) and first public word (14:12): the entire blast window had nearly closed before the status page acknowledged anything. Second, the mitigation was tenant-side — "redeploy or rollback to a previous deployment" — which means the tenants who could have fixed themselves fastest were the ones least likely to know, since nothing told them their tier was the blast radius until the postmortem-style resolution note named the 10:18–14:10 window explicitly.
Credit where due: the resolution update did name the exact window and the exact scope ("a subset of deployments created under the Hobby plan teams"), which is more precise than most status prose. The failure was not dishonesty; it was latency. A tier-scoped failure needs tier-scoped detection, and the detection arrived four hours after the tenants did.
The six-minute telemetry hole, disclosed the next day
The July 23 incident, "Telemetry data loss for Drains", is the more consequential of the two despite lasting six minutes — because what tenants lost was not uptime but the evidence of uptime. Between 19:12 and 19:18 UTC, telemetry forwarded through Drains was not delivered. Vercel Drains is the unified pipeline that streams logs (runtime, build, static, firewall, function), OpenTelemetry traces, events, and analytics out to tenants' own tools — Datadog, Sentry, OTLP endpoints — where tenants build their "status >= 500" alerts and their audit trails. A gap in that pipeline is a gap in every downstream alert, dashboard, and compliance export at once.
The loss was asymmetric, and the asymmetry matters:
| Signal | Fate of the 19:12–19:18 window |
|---|---|
| Traces forwarded via Drains | Dropped, unrecoverable |
| Events forwarded via Drains | Dropped, unrecoverable |
| Logs forwarded via Drains | Not delivered — but still available in the Logs UI dashboard, and exportable from there |
Unrecoverable trace and event loss is the severe kind of observability failure: there is no retry, no backfill, no dashboard to re-export from. Any tenant whose incident review, anomaly detection, or audit evidence depended on those six minutes of traces simply has a hole. Vercel rated the incident major — higher than the Hobby incident's minor — which reads as the company agreeing: silent data loss outranks loud downtime.
Then there is the disclosure timeline. The incident record was filed retroactively (created and resolved both stamped July 23, 19:00 UTC) with a single update posted July 24 at 17:26 UTC — roughly 22 hours after the window. No investigating/identified/monitoring trail exists because there was nothing to investigate in real time; the failure was discovered after the fact. Vercel's own closing line concedes the detection gap: "we have deployed a fix and are adding additional monitoring to detect and prevent this failure mode from happening in the future." And a telling metadata detail: the incident record lists zero affected components — the pipeline every tenant's external observability depends on had no component identity on the status page at all.
The same shape, twice more in the same quarter
Scoped failures were not a July anomaly; Vercel's own status history shows the pattern repeating within weeks, which is what elevates this from "two bad days" to a structural property of multi-tenant control planes:
- August 6 — "Delays delivering log drains" (minor). The same forwarding pipeline, two weeks later: drain delivery delayed from 16:37 UTC, recovering by 18:19, fully resolved with the backlog processed by 18:29. This time nothing was lost — delayed, then caught up — but it confirms the drain path as a repeated scoped-failure surface, now with a live incident trail instead of a retroactive one.
- September 1 — "Increased deployment failures" (major). "Some deployments that use IAD1 Function regions or Routing Middleware are failing" — a region-plus-feature-scoped blast radius, 19:59 to 20:58 UTC. Same shape as the Hobby incident with different axes: not one tier but one region and one middleware path.
Side by side, the four incidents share one structure:
| Incident | Blast-radius scope | Detection/disclosure lag | Who could notice first |
|---|---|---|---|
| Jul 17 Hobby invocations | One plan tier, new deployments only | ~4h (10:18 impact → 14:12 post) | Affected Hobby tenants, via their own errors |
| Jul 23 Drains telemetry loss | One forwarding pipeline, all drain users | ~22h, retroactive, no live trail | Nobody in real time — the gap was silent |
| Aug 6 drain delays | Same pipeline | Live trail, ~2h to full recovery | Tenants watching drain lag |
| Sep 1 deployment failures | One region + one middleware path | Live trail, ~1h | Tenants deploying to IAD1 |
The general lesson is one the status-page ecosystem keeps relearning: a page that reports platform health cannot represent slice health. As the open-source OpenStatus guides put it, "a status page that shows 'all systems operational' during an obvious outage destroys trust faster than no status page at all." Spotify relearned the same lesson the same month, when its status page stayed silent through a July 2026 outage that left users guessing. Self-reported, platform-granularity status is structurally blind to exactly the failures multi-tenant platforms produce most often.
What a deploy platform should guarantee instead
The design question these incidents pose is precise: a scoped failure should either surface to the affected tenant immediately or not happen invisibly at all. Four concrete guarantees follow from the four incidents above:
1. A tenant-scoped health signal alongside the platform status. The Hobby incident's four silent hours happened because "Functions" as a platform component looked fine while "Functions for new Hobby deployments" did not. Any deploy API already knows the tenant's plan tier, region, and feature flags — the status surface should answer "is my slice healthy?" with the same prominence as "is the platform healthy?" For agent-operated infrastructure this needs to be machine-readable, not a page to scrape.
2. Deliver-or-dead-letter semantics with lag/loss alerting on telemetry pipelines. The Drains incidents are a pipeline-reliability problem wearing an observability costume. A forwarding pipeline should treat undelivered telemetry the way a queue treats unacked messages: buffer it, retry it, dead-letter it — and page when lag or loss crosses a threshold, before a tenant's audit window closes. August 6 (delayed, then caught up) is what this guarantee looks like when it works; July 23 (dropped, discovered next day) is what its absence costs.
3. Component and scope attribution on every incident. The Drains data-loss record shipped with zero components attached and a retroactive timestamp — the worst possible combination for anyone doing blast-radius triage ("were we affected?"). Every incident needs the two fields Vercel's better updates already include informally: which component and which slice (tier, region, feature path, time window). Make them mandatory fields, not prose conventions.
4. Retroactive disclosure of data-loss windows even when already "resolved." Vercel got this one right and it deserves to be a norm: the July 23 entry was posted even though there was nothing left to fix, because tenants needed to know their trace history had a hole. "Resolved" must never mean "not worth mentioning" when the failure destroyed tenant data rather than merely delaying it.
The platform is fine; your slice is the product
Multi-tenant platforms fail in slices — one tier, one pipeline, one region — far more often than they fail wholesale, and every layer of status infrastructure built for wholesale failure goes quiet at exactly the wrong moment. Vercel's July is instructive not because the incidents were severe but because they were ordinary: a four-hour tier-scoped compute failure and a six-minute feature-scoped observability gap, both real, both disclosed, both shaped like the failures any Render-compatible deploy API will produce on its own control plane.
The forward-looking version of this problem is sharper. As AI agents become the operators — deploying, watching dashboards, triaging alerts — "a human might notice the status page eventually" stops being a backstop at all. Agents need scoped, machine-readable health ("my tier, my region, my pipeline, my window") the way they need typed API errors: without it, they will do what every Hobby tenant did for four hours on July 17 — stare at failing invocations while the platform insists everything is fine.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
