The Deploy That Pointed at Nothing: Firetiger's CI Race Condition and the One Check That Would Have Stopped It

At 08:30 UTC on February 28, 2026, a routine concurrency rule canceled a CI build in the middle of a merge. Five minutes later, a deploy pipeline that had no idea the build was dead updated Firetiger's ECS and Lambda service definitions to point at a container image that was never actually built. The system kept running fine for the next 25 hours — old containers don't care what your service definition says. Then, at 09:40 UTC on March 1, an ECS task exited normally, tried to restart, and couldn't. Neither could any task after it. For eight hours, Firetiger's ingest pipeline returned 503s on OpenTelemetry data and GitHub webhooks, because every replacement task pointed at an image ID that didn't exist anywhere in the registry.

That's the whole failure in one paragraph, and it's worth sitting with, because nothing about it required a sophisticated attack or exotic bug. A build got canceled. A deploy job trusted a signal instead of checking a fact. A scheduler kept pointing at nothing until the last thing that was actually running fell over. Firetiger's own postmortem is unusually candid about the mechanism, and it maps directly onto a question every git-push deploy pipeline has to answer honestly: when your deploy step writes an image ID into a manifest, does it know that image exists, or does it just trust that the step before it said so?

How a canceled build became a real deploy​

The failure has two independent causes, and neither one alone would have caused an 8-hour outage. Here's the timeline, in UTC:

The first bug — a canceled build — is mundane. Concurrency rules that cancel superseded CI runs are standard practice; GitLab's own issue tracker has multiple open reports of exactly this class of race, where two runs interact in an order the pipeline didn't anticipate. The second bug is the one that actually matters: the deploy pipeline had a way to be wrong about whether a build had finished, and nothing downstream double-checked it before writing the result into infrastructure.

The 25-hour bomb: why "it worked yesterday" told them nothing​

The most dangerous part of this incident isn't the two bugs — it's the 25 hours in between, when everything looked fine. The bad ECS service definition was live from February 28 onward, but ECS doesn't proactively validate a task definition against the registry when nothing is changing. It only finds out an image is missing when it actually needs to launch a task against that definition — a deploy, a scale-out event, or, in this case, a task exiting and needing a replacement.

That's a structural property of most container orchestrators, not a Firetiger-specific flaw: a broken reference sits inert until something forces a fresh pull. Which means the actual failure mode isn't "the deploy was bad" — it's "the deploy was bad, and there was no moment between the bad deploy and the eventual outage where anyone or anything was forced to notice." The system had 25 hours of silent invalid state before the first ECS task happened to exit and expose it. A CI/CD system that treats "the deploy succeeded" as the end of the verification chain has no way to catch a fault like this until production traffic finds it — at a time nobody chose, on a task nobody was watching.

Terraform caught half the problem — and that half-catch made it worse​

There's a subtler failure buried in the Terraform step that's easy to miss on a first read: Terraform did catch something wrong. The Lambda update referenced the same nonexistent image, and Terraform rejected it as invalid, aborting the deployment. If that had been the only resource in play, the story ends there — a failed terraform apply, an alert, a human looks at why.

But Terraform doesn't apply a plan atomically across every resource in it. It had already mutated the ECS service definition in an earlier step of the same apply before it got to the Lambda resource and bailed. The abort stopped the rest of the damage, but not the part that already happened — leaving ECS in exactly the half-broken state that produces a delayed outage instead of an immediate, loud one. A clean failure (both resources rejected, or both applied) would have either avoided the incident entirely or caused a same-day, easily attributable one. The partial failure did neither: it produced a system that looked deployed, passed no immediate alarms, and quietly wired production traffic to fall over on the next unrelated task restart.

The one check that was missing​

Strip away the specific vendor names and Firetiger's incident reduces to a single design gap: the deploy pipeline trusted a build-completion signal instead of verifying the artifact the signal claimed to reference. That's a fixable, well-understood problem, and the fix doesn't require redesigning CI:

1. Pin by digest, not tag. A tag (myapp:latest, myapp:build-4821) is a mutable pointer that a CI bug can point at the wrong thing. A digest (myapp@sha256:...) is a content hash — it either exists in the registry or it doesn't, and there's no ambiguity for a downstream bug to exploit. Most registries support resolving a build to its digest as the very last CI step, before anything downstream ever sees a tag.
2. Check existence at the moment of rollout, not at the moment of build. Before a deploy step writes an image reference into a service definition, it should perform one cheap registry call — a HEAD request or a docker manifest inspect/crane digest equivalent — against that exact digest, right then. Not "the build reported success five minutes ago." Right now, against the registry that will actually be asked to serve the pull later.
3. Fail the rollout, not just the resource. If that check comes back empty, the deploy should refuse to touch any resource in the rollout — the Terraform half-failure in this incident is exactly what happens when a verification gap lets a multi-resource apply get partway through before something finally objects.

None of this is exotic. Industry guidance on digest pinning has been converging on exactly this pattern for reasons unrelated to this specific incident — reproducibility, tag-mutation attacks, audit trails. Firetiger's outage is a concrete demonstration of the failure mode that guidance exists to prevent: an image reference a deploy pipeline never independently verified, sitting in a service definition, waiting for the moment something actually needs it to be real.

An agent found it in minutes — after the fact​

The postmortem includes a detail worth calling out on its own: Firetiger's team resolved the incident by pointing Claude Code at the issue report. Using an MCP connection along with AWS and GitHub CLI access, the agent located the broken deployment and identified the missing image reference as the root cause.

That's a genuinely useful data point about what an agent with the right tool access can do during incident response. But it also sharpens the question this whole incident raises: if an agent with registry and cluster access can determine "this image ID doesn't exist" in a matter of minutes after an 8-hour outage has already happened, that's not a capability that should be reserved for postmortems. The exact check the agent ran manually — does this image exist, right now, in the registry the cluster will actually pull from — is a check a deploy pipeline can run automatically, in milliseconds, before the rollout ships. The value of an agent finding the answer fast after the fact is real; the value of not needing to ask the question after the fact at all is bigger.

What this means for a deploy step you actually own​

A git push-triggered deploy pipeline has the exact same shape as the one that failed here: a build stage that reports success or failure, and a deploy stage that trusts that report and writes infrastructure based on it. The gap Firetiger hit isn't specific to Terraform, ECS, or their particular concurrency bug — it's specific to any pipeline where the deploy step's only source of truth about the artifact is a signal from an earlier, separately-failing step.

The fix belongs in the deploy step itself, not in trying to make the build step unable to lie. A deploy step that resolves the build to its content digest and confirms that digest is pullable from the registry immediately before handing a manifest to the scheduler closes exactly the gap Firetiger's postmortem describes — regardless of what upstream race condition, concurrency bug, or artifact-attribution error produced a false "build succeeded" signal in the first place. You don't have to predict every way the signal can be wrong. You just have to stop trusting it and check the thing it's claiming instead.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with a deploy step that verifies the image it's about to run actually exists before handing it to the scheduler. Star the repo on GitHub or deploy your first app today.

Sources​

• Firetiger: Incident postmortem — the March 1, 2026 ingest incident.
• GitLab: Pipeline Race Condition could change correct deploy order.
• OneUptime: How to Implement Image Digest Pinning for Immutable Kubernetes Deployments.