Fly.io's Logs Went Dark for 4.5 Hours While Apps Stayed Up: What Owning Your Log Pipeline Actually Costs

On October 13, 2026, at 09:25 UTC, Fly.io's apps kept serving traffic and its status page stayed reassuringly free of red. But if you opened a historical log query any time in the next four and a half hours, the past simply was not there. The incident — titled, with admirable understatement, "Historical logs are currently delayed" — ran until 13:50 UTC, a 4-hour-24-minute window in which the platform was up and your visibility into it was not (Pingoru incident record).

Fly's own status message named the mechanism plainly: "Our log ingestion pipeline is currently backed up. Historical logs will be delayed while the queue is processed." Your applications were fine. The queue between your applications' output and the search index you query had filled up, and everything behind it waited. There is a special helplessness to this failure mode: nothing to restart, no region to fail over to, no fix on your side at all. You just could not see.

And this was the second time in thirteen days. On October 1, Fly posted "Delayed ingestion of app logs," investigating delays "across all regions" and warning that "new logs from machines may not be visible." Two ingestion-queue incidents in under two weeks is not a fluke; it is the failure mode of a log path you do not own, arriving on schedule.

This post is about that failure mode — which half of the pipeline it lives in, what it costs you when it lands mid-incident, and what it concretely costs to own the log path instead so the next backed-up queue is one you can see, alert on, and drain yourself.

Which half of the pipeline failed​

To understand what went dark, you need Fly's log anatomy. Each Fly Machine captures stdout/stderr, ships those logs over NATS, and stores them "for a period of time in a VictoriaLogs-backed search index," in Fly's own description. Most users consume logs through fly logs or by shipping them to an external sink.

Those two consumption paths failed very differently on October 13. fly logs is a live tail backed directly by the NATS stream — low-latency, real-time, no index involved. The historical search path is the VictoriaLogs index behind an ingestion queue, holding roughly the last seven days of logs.

When the ingestion pipeline backs up, the live tail keeps flowing while the past freezes. Anything you needed to look back at — the deploy that went out at 09:00, the error spike at 10:30, the request trace from the customer ticket filed at noon — sat behind an unprocessed queue for up to four and a half hours.

That asymmetry is what makes this the worst kind of green dashboard. A full outage pages you, focuses the team, and triggers the runbook. A log-visibility gap pages nobody: your uptime monitor is green, your health checks pass, and the one tool that could tell you why the checkout error rate tripled an hour ago returns stale results with no error banner. You cannot tail what has not been delivered, and during those 264 minutes, nothing historical was being delivered.

The honest scope of the damage is bounded, and worth stating plainly. Live debugging of a currently-burning issue still worked through the NATS tail. What died was forensics: post-deploy verification against historical baselines, correlation across services over a time window, and any alert or dashboard built on the managed search index rather than the live stream. If your incident happened to need the past — and most interesting incidents do — you were debugging blind until 13:50 UTC.

What your logs cost everywhere​

Here is the table this post owes you. Take a typical small workload — a handful of services emitting about 10 GB of ingested logs per month — and price it across the managed sinks Fly users actually ship to, plus the owned alternative, at 10 GB and at 100 GB per month to show how each option scales.

Sources: Datadog list prices as of September 2026 and the 4.81M-events-per-GB conversion (devops-daily teardown); Better Stack and Axiom plan figures (Datadog-alternatives pricing survey, observability-on-a-budget guide); Hetzner Cloud 2026 pricing for the CX22 at roughly €4.35/month after the April 2026 price revision.

Three things stand out. First, Datadog's bill is almost entirely the indexing line — the $0.10/GB ingest charge is a rounding error next to ~$81.80 in per-event indexing at 10 GB, and that ratio holds as you scale. Second, the budget-managed tier (Better Stack, Axiom) has collapsed the price of someone-else-operates-it logging to single dollars at small volumes — the "just ship to a vendor" answer has never been cheaper. Third, the owned box is flat: €4.35 whether you ingest 10 GB or 100 GB, and with ~10x compression a 40 GB disk holds months of a 10 GB/mo workload — far past Fly's own ~7-day managed retention window.

But the table is only half the argument, and the cheaper half. The expensive half is the October 13 question: when the managed pipeline backs up, whose queue is it, and can you see it? Every row above except the last one moves the queue without giving you visibility into it. Axiom and Better Stack run excellent pipelines, but their ingestion lag is their status page, not your metric. Only the owned row changes who gets to watch the queue drain.

Owning the path: the stack and the on-call​

Owning the log path does not mean building a logging company. The minimal serious stack has three parts, and Fly's own ecosystem already demonstrates the pattern.

Subscribe at the source, don't poll the index. Fly exposes its platform log flow as a NATS stream at nats://[fdaa::3]:4223, and the official log-shipper guidance is a Vector-based app that subscribes to that stream and forwards to a sink. The architectural lesson generalizes beyond Fly: tap the firehose where it is emitted — Vector, Alloy, or OpenTelemetry Collector on your own fleet — rather than depending on a managed search index as your only copy. Fly even publishes fly-telemetry, a reference stack that subscribes to the platform NATS streams and writes logs and metrics to local VictoriaLogs and VictoriaMetrics behind your own Grafana. The vendor's own answer to "what if our index lags" is, notably, "run your own copy of the stream."

Store it in a single binary that fits a small box. This is where the economics flipped in the last few years. VictoriaLogs and Loki both run as single binaries with no required external dependencies, and independent benchmarks put VictoriaLogs at roughly 1.3 GB of RAM and near-idle CPU on the same workload where Loki holds 6–7 GB (500 GB benchmark), with typical compression around 10x. Either one fits comfortably on a €4–8 Hetzner box alongside Grafana; VictoriaLogs additionally speaks a Loki-compatible push endpoint, so existing collectors migrate without relabeling. A homelab-scale reference stack reports ~10 GB of disk holding its Loki retention comfortably — on a 40 GB CX22, disk is simply not the constraint at small-team volumes.

Make ingestion lag your metric, not their status post. This is the actual point. When you own the collector-to-store path, the queue that backed up on October 13 becomes a thing you can instrument: consumer lag on the NATS subscription, Vector buffer depth, store ingestion rate versus arrival rate. You alert on lag exceeding minutes; you watch it drain after a burst deploy; when it backs up, you scale the consumer or shed low-value streams yourself instead of waiting for "while the queue is processed." The failure mode does not disappear — queues still fill — but it converts from a black box with a status page into a dashboard panel with a runbook.

Now the honest accounting, because the owned row in the table has costs the price column hides. You operate retention (set it, enforce it, handle the disk-full case before it pages you at 3 AM). You operate upgrades for the store, the collector, and Grafana. You are the person who notices when a noisy service's debug logging 10x's your ingestion overnight — the managed vendors, for all their faults, absorb that burst without waking you.

And your single €4.35 box is a single point of failure until you give it a second disk, a backup job, or a replica, each of which costs more than the box. Owning the path trades a per-GB meter and a blind queue for a flat hardware bill and a small, permanent operations burden. For a team already running its own fleet, that burden rounds to zero. For a team that chose a PaaS specifically to never think about infrastructure, it is real — price it honestly before you commit.

The platform is up; are you sure?​

The October 13 incident will not be the last of its species. Every managed log pipeline is an ingestion queue in front of a search index, every queue can back up, and the vendor's status message will always read some variant of "delayed while the queue is processed." The question is never whether the queue exists. It is whether you can see it.

If you run on Fly — or Render, Railway, or any PaaS whose historical logs live behind someone else's ingestion — the cheapest insurance is a second copy of the stream you control: subscribe at the source, land it in a store on hardware you own, and keep your forensics available on the day the managed index goes quiet. Four and a half hours of blindness, twice in thirteen days, is the price of the alternative.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.