Let's Encrypt's 2.5-Hour Outage Broke Live Renewals: A Real Fallback-CA Design for Self-Hosted ACME

At 18:37 UTC on May 8, 2026, Let's Encrypt's production ACME endpoint started returning HTTP 503 to every certificate request that hit it. Not a slowdown, not elevated error rates — a hard stop. For the next two and a half hours, every certbot cron, every acme.sh daemon, every cert-manager reconcile loop, every cPanel AutoSSL job anywhere in the world that happened to fire during that window got the same answer: no certificate for you.

Two and a half hours doesn't sound like an incident worth an architecture change. That's exactly the assumption this post is going to take apart — because the honest answer is "it depends entirely on what your renewal buffer looks like," and Let's Encrypt's own roadmap is shrinking that buffer for everyone, on purpose, right now.

The outage, in numbers​

Here's what actually happened, reconstructed from Let's Encrypt's incident history and postmortem:

The root cause was narrow: the cross-signed subordinate CAs bridging the old Generation X root to the new Generation Y root were missing an Extended Key Usage field that newer trust-store policy now requires. Root-cause fixed, rollback executed, service restored in roughly 2 hours 28 minutes. As outages go, this was handled about as well as an outage can be handled.

It also wasn't a one-off. It's the third publicly logged Let's Encrypt ACME API outage in twelve months:

None of these is evidence Let's Encrypt is badly run — a service issuing certificates for over half the web at zero cost to any of them is going to have incidents, and this one's postmortem is a clean read. The pattern is the point: an outage on the order of an hour to a few hours happens roughly every four to six months, and any automation that treats "Let's Encrypt is always up" as an assumption baked into its renewal design is going to get tested on that cadence whether it planned for it or not.

What actually broke, and what didn't​

Let's Encrypt's own incident notes are direct about the immediate blast radius: any renewal job — certbot, acme.sh, Caddy, Traefik, cPanel AutoSSL, a managed host's own renewal daemon, cert-manager — that ran against the tlsserver or shortlived ACME profiles during the 18:37–21:03 window got an ACME error. That's not a hypothetical; it's every in-flight renewal attempt in the world for those two-and-a-half hours.

Most of those recovered on their own. Certbot's systemd timer fires twice a day; acme.sh runs daily; Caddy retries on its own backoff. A renewal that failed at 19:00 UTC and got picked back up by the client's next scheduled attempt the following day never became an operator's problem. That's the case Let's Encrypt's own guidance leans on: "if your ACME client honours [ACME Renewal Information], the next scheduled run will see the renewal hint and act on it — you don't need to do anything."

Two categories of renewal did not self-heal that cleanly:

1. Certificates issued by the affected profiles between the initial halt and the May 13 reissue. Their chain still pointed at the defective cross-sign until Let's Encrypt corrected it five days later. Let's Encrypt's own recommendation was to actively re-renew them rather than wait — a step a client that doesn't poll ARI, or an operator who wasn't watching the incident page, would simply miss.
2. Any renewal running on a schedule tight enough that the outage window ate the whole remaining buffer, rather than a fraction of it. This is the case worth designing for, and it's the one that gets categorically worse as Let's Encrypt's own roadmap plays out.

The buffer math: why the same 2.5 hours means two different things​

The question that actually matters for a platform operator isn't "did Let's Encrypt have an outage" — it's "how much of my renewal safety margin did that outage consume." That number depends entirely on your certificate's lifetime and your renewal cadence, and it changes by more than an order of magnitude between Let's Encrypt's standard profile and its short-lived one.

Standard 90-day certificates, renewed at the common default of 30 days before expiry, carry a renewal buffer of 720 hours. A 2.5-hour outage consumes about 0.35% of that buffer — genuinely negligible, which is why the "the next scheduled run will succeed" guidance holds for almost everyone.

Let's Encrypt's shortlived profile, generally available since January 2026, issues certificates valid for 160 hours (just under 7 days). ACME Renewal Information suggests renewing with roughly a 48-hour buffer before expiry on that profile. A 2.5-hour outage against a 48-hour buffer consumes about 5.2% of it — fifteen times the proportional hit a standard certificate takes from the identical outage.

5.2% of a safety margin surviving one outage is still fine on its own. It stops being fine the moment a second variable stacks on top: a renewal job that only runs once a day instead of continuously retrying. A nightly cron sweep was a completely reasonable design at 90-day cadence — you had weeks of slack even if one run silently failed. At a 48-hour buffer, a single missed daily run isn't "you'll catch it tomorrow," it's "you've now burned half your entire safety margin on one skipped cycle," and a second miss — the outage landing on the one day your nightly job happened to fire during the 503 window — can push the renewal attempt past the point where the certificate has already expired before anyone notices.

This isn't a shrinking edge case. Let's Encrypt has published a staged roadmap taking the default certificate lifetime from 90 days down to 45 by 2028. Every step of that roadmap tightens the exact ratio in the table above for every certificate on the platform, not just the ones that opted into shortlived. The renewal automation that was comfortably over-provisioned in 2026 is the renewal automation that runs out of margin by default in 2028, on a timeline Let's Encrypt already committed to publicly.

cert-manager has no built-in fallback — here's what to build instead​

If you're running a self-hosted PaaS on Kubernetes, cert-manager is almost certainly issuing your tenant TLS certificates, and it's worth being precise about what it does and doesn't give you here: cert-manager has no automatic failover between ACME issuers. A ClusterIssuer points at exactly one ACME endpoint, and if that endpoint is down, the Certificate resource sits in a failed state and retries against the same issuer until it recovers. There's no config flag that says "try a second CA if the first one is unreachable" — you have to build that yourself.

The pieces aren't exotic. Run two ClusterIssuer resources:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-primary
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@yourdomain.com
    privateKeySecretRef:
      name: letsencrypt-primary-key
    solvers:
      - dns01:
          # your DNS-01 solver config
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: gts-fallback
spec:
  acme:
    server: https://dv.acme-v02.api.pki.goog/directory
    email: ops@yourdomain.com
    externalAccountBinding:
      keyID: <your-gts-eab-key-id>
      keySecretRef:
        name: gts-eab-secret
        key: eab-hmac-key
    privateKeySecretRef:
      name: gts-fallback-key
    solvers:
      - dns01:
          # same DNS-01 solver config

Then add the failover logic cert-manager itself doesn't provide: a small controller (or a scheduled job) that watches Certificate resources for renewal attempts that have failed past a threshold — say, more than N consecutive failures, or age within a fixed window of the certificate's actual expiry — and patches spec.issuerRef to point at gts-fallback instead of letsencrypt-primary. cert-manager picks up the change and re-issues against the second CA on its next reconcile. That's the entire mechanism: two issuers, one trigger condition, one patch. Nothing about it requires exotic infrastructure — it requires deciding, in advance, what "failed enough to fail over" means for your fleet, instead of discovering the answer during the next multi-hour outage.

Picking the second CA​

Not every free ACME CA is a realistic fallback candidate in 2026. Buypass discontinued its Go SSL service in October 2025 — no new orders, no renewals — which removes it from consideration entirely despite still showing up in older "Let's Encrypt alternatives" writeups. That leaves two credible options:

For most self-hosted PaaS operators, ZeroSSL is the more practical default fallback — the EAB setup is a single API call away, with no separate cloud project or billing relationship to provision and maintain just to keep a break-glass CA warm. Google Trust Services is the stronger choice if you're already deep in GCP and the extra project is free marginal overhead rather than a new dependency. Either beats the alternative, which is having no second issuer configured at all and finding out during the next outage.

Alert on hours left, not on "expired"​

The other half of surviving an outage like this is noticing before the certificate actually expires, not after. "Alert when the certificate has expired" is not an alert — it's an incident report written by the outage itself, arriving with zero lead time to do anything about it. The threshold has to fire while there's still runway to act, whether that's forcing a manual renewal, kicking the failover trigger above by hand, or just paging someone before a tenant's site goes down with a browser TLS warning.

That threshold can't be a flat number of days, for the same reason the outage's impact isn't flat across profiles: a fixed "alert when 7 days remain" rule is generous slack on a 90-day certificate and already-too-late guidance on a 160-hour one, where 7 days exceeds the certificate's entire lifetime. Scale the alert to the same ratio the buffer math above uses — a percentage of the certificate's total lifetime remaining unrenewed, not an absolute day count — so a shortlived certificate pages someone at, say, 15% of its lifetime left (roughly 24 hours) while a 90-day certificate pages at the equivalent 15% (about 13 days). Same rule, correctly scaled, instead of one threshold quietly meaning two very different things depending on which profile issued the certificate.

Why this is worth building before the next outage, not after​

Let's Encrypt issues something like 54–63% of all publicly trusted TLS certificates on the web today, depending on the measure. Add Google Trust Services and Amazon Trust Services — the other two free, fully automated CAs — and the top three collectively account for over three-quarters of everything issued. That's not a knock on any of them; it's the natural result of "free and automatic" winning the market it was built for. It does mean that a meaningful fraction of the entire web's TLS renewal automation has, by default, exactly one CA it knows how to talk to — and every operator running that default found out what that costs them for two and a half hours on May 8, 2026, and will find out again, on a roughly four-to-six-month cadence, going forward.

A self-hosted, git-push PaaS that issues and renews every tenant's custom-domain certificate itself already owns this problem completely — there's no CDN or managed-TLS vendor absorbing it on your behalf. That's also exactly the position from which the fix costs the least: a second ClusterIssuer, a failover trigger, and a lifetime-scaled alert threshold are Kubernetes-native primitives you're already running, not a new product tier or a vendor contract. The platforms that treat single-CA dependency as a design decision, rather than an accident of "the tutorial only configured one ClusterIssuer," are the ones whose tenants don't notice the next acme-v02.api.letsencrypt.org 503 at all.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, custom domain and TLS included. Its own ACME automation runs a second issuer as a standing fallback rather than a break-glass afterthought, because a platform that owns tenant TLS doesn't get to treat "the CA is always up" as someone else's assumption. Star the repo on GitHub and see how the failover path is wired before your own renewal job finds out the hard way.