Skip to main content

Ingress-NGINX Is Retired: A Zero-Downtime Migration Plan for Every Tenant Domain on Your Git-Push PaaS

11 min readDora NodaDora Noda
Share
On this page

In March 2026, the most widely deployed ingress controller in Kubernetes history stopped receiving security patches. Ingress-NGINX — the controller behind countless nginx.ingress.kubernetes.io annotations — was retired by SIG Network and the Security Response Committee; best-effort maintenance ended in March 2026. Your existing deployments still run. The installation artifacts are still there. But from now on, every CVE discovered in that edge proxy is yours to live with, unpatched, on the exact component that terminates every byte of tenant traffic.

If you operate a git-push PaaS, this is not a someday problem. Your routing layer sits in front of every tenant domain at once — the generated *.yourplatform.com wildcard and every custom domain a tenant ever pointed at you. This post is the migration plan: translate with Ingress2Gateway 1.0, shadow-validate the output, cut over host by host on weighted DNS, and roll back on thresholds instead of hope. The whole plan in one table:

StepActionProof it worked
1. InventoryList every Ingress and every NGINX annotation in the fleetA complete annotation census, no surprises in step 3
2. InstallDeploy Gateway API CRDs (Standard channel, v1.5+) and your chosen Gateway implementation alongside ingress-nginxkubectl get gatewayclass shows your implementation, prod untouched
3. TranslateRun ingress2gateway print per IngressGateway + HTTPRoute YAML plus a warning log
4. TriageClassify every warning: clean, best-effort, unsupportedZero unclassified warnings
5. ShadowApply Gateway/HTTPRoutes next to live Ingress; mirror and diff real responsesByte-level equivalence on status, headers, body for sampled traffic
6. Cut overShift one host at a time with weighted DNS; wildcard fleet first, custom domains secondError-rate and latency dashboards flat across each shift
7. DecommissionDelete Ingresses, uninstall ingress-nginxNo nginx.ingress.kubernetes.io annotation left in the fleet

Note what step the YAML translation is: step 3 of 7. A successful translation is a starting draft, not proof. Everything after it exists because routing behavior is what live traffic experiences, not what a converter emitted.

Two tracks: wildcard first, custom domains second

A PaaS has two very different domain populations, and the migration treats them differently:

Track A: generated wildcardTrack B: tenant custom domains
Exampleacme-app-42.yourplatform.comapp.acme.com (tenant-owned DNS)
Annotation mixUniform — your deploy pipeline generated them allHeterogeneous — years of per-tenant CORS, rewrite, auth-snippet accretion
TLSOne wildcard cert, operator-managedPer-tenant certs via cert-manager, HTTP-01 or DNS-01
DNS controlYours — you can shift weights freelyShared with the tenant — TTLs and expectations vary
Role in planThe canary: migrate first, prove the pathThe bulk: migrate host-by-host after Track A is clean
Rollback blast radiusWhole fleet at once (so validate hardest here)One tenant at a time

Track A goes first precisely because it is uniform and fully under your control: if the translated HTTPRoutes misbehave, you see it on your own domains before any tenant does. Track B then proceeds host by host, slowest-first for the tenants with the strangest annotation sets.

Run the runbook

Step 1 — inventory. Before touching anything, census what you actually run. Every Ingress object, every nginx.ingress.kubernetes.io/* annotation, grouped by host:

bash
kubectl get ingress -A -o json | \
  python3 -c "import json,sys; [print(i['metadata']['namespace'], i['metadata']['name'], sorted(i['metadata'].get('annotations',{}))) for i in json.load(sys.stdin)['items']]"

The output you want is a list of distinct annotations. That list is the input to triage in step 4 — and it tells you which tenants to migrate last (whoever uses the rarest annotations).

Step 2 — install alongside. Install the Gateway API Standard-channel CRDs (v1.5, released February 2026, is the stability release — TLSRoute, HTTPRoute CORS filters, ListenerSet, client-certificate validation, and ReferenceGrant all graduated to Standard) and your chosen Gateway implementation in the same clusters. More on that choice below. Do not remove ingress-nginx. Both data planes run; only one serves production traffic.

Step 3 — translate. Ingress2Gateway 1.0, announced March 2026 by SIG Network, is the official migration assistant: it converts Ingress resources plus provider-specific annotations into Gateway API manifests, and warns about whatever it cannot carry over. Where pre-1.0 builds understood three ingress-nginx annotations, 1.0 supports over 30, each backed by controller-level integration tests that compare runtime behavior (routing, redirects, rewrites) in live clusters — not just YAML shape:

bash
ingress2gateway print \
  --input-file tenant-ingress.yaml \
  --providers ingress-nginx > gwapi.yaml

You get a Gateway with per-host listeners, one HTTPRoute per host, and — critically — a warning log. In the project's own worked example, a single Ingress with CORS, regex paths, timeouts, body-size, and a configuration snippet produced four distinct warnings. That log is the real output of this step; the YAML is a draft.

Step 4 — triage every warning. Sort the log into three buckets:

BucketExamplesDisposition
Translates cleanlyenable-cors → CORS response-header filter; rewrite-target → URLRewrite filter; regex path → RegularExpression match; TLS block → HTTPS listener with certificateRefsAccept, then verify in shadow
Best-effort with a warningproxy-read-timeout / proxy-send-timeouttimeouts.request (ingress-nginx timeouts are TCP-level; Gateway API timeouts are request-level — related, not identical); regex emitted as case-insensitive prefix (?i)/users/(\d+).* (you usually want to strip the (?i) and trailing .*)Hand-edit the generated route, note the deviation
Unsupportedconfiguration-snippet (arbitrary NGINX config has no Gateway API equivalent); proxy-body-size (no standard field — you inherit your implementation's defaults); URL normalization (varies by implementation, not configurable via standard API)Reimplement in your Gateway's native policy CRD, or change the app to not need it

The unsupported bucket is where migrations die quietly. configuration-snippet is the big one: it is raw NGINX config — Lua blocks, auth subrequests, header surgery — and no converter can express it portably. Each instance needs a human who understands what the snippet did and how your chosen implementation reproduces it (Envoy Gateway's BackendTrafficPolicy, Traefik middlewares, Cilium policies). Tenants with snippets migrate last and get individual attention.

Step 5 — shadow-validate with live traffic. Apply the Gateway and HTTPRoutes while ingress-nginx still serves production. Then prove equivalence, not plausibility: replay sampled requests at both data planes and diff. The cheap version is a Host-header loop against the new Gateway IP:

bash
GATEWAY_IP=$(kubectl get gateway platform -o jsonpath='{.status.addresses[0].value}')
for host in app-42.yourplatform.com app.acme.com; do
  for path in / /healthz /api/users/123; do
    diff <(curl -s -o /dev/null -D - -H "Host: $host" "https://prod-ingress-ip$path") \
         <(curl -sk -s -o /dev/null -D - -H "Host: $host" "https://$GATEWAY_IP$path") \
      && echo "MATCH $host$path" || echo "DIFF $host$path"
  done
done

Compare statuses, redirect chains, rewritten paths, CORS headers, and bodies — not just 200-vs-200. Keep ingress-nginx authoritative until the diffs are clean across every host in the current batch, including the timeout and body-size edge cases from the triage table.

Step 6 — cut over host by host on weighted DNS. This is the zero-downtime mechanism, and it is DNS weights, not DNS swaps. Publish both data planes behind weighted records, shift weight gradually to the Gateway, watch error rate and p99 latency per host, and reverse the weights on any regression. Track A (your wildcard) shifts first as the fleet-wide canary; Track B follows tenant by tenant. The CNCF-documented production playbook for this exact migration (ingress-nginx to Envoy Gateway) used precisely this pattern — parallel operation with weighted Route 53 records and continuous polling verification — because weight reversal is instant while a swapped record waits out TTLs.

Define the rollback triggers before you start: per-host 5xx rate above baseline plus a margin, p99 latency regression beyond an agreed bound, or any TLS handshake failure. A rollback is a weight change, not an incident.

Step 7 — teach the deploy pipeline, then decommission. On a git-push PaaS, Ingresses are generated artifacts: every push mints or updates routing objects. If the generator still emits Ingress on the day you finish migrating tenants, the next push recreates the old world. The generator must learn to emit HTTPRoute (attached to your Gateway via parentRefs) instead — and you must backfill every existing tenant Ingress through the same code path so hand-migrated YAML and generated YAML never diverge. Only then: delete the Ingress resources, uninstall the ingress-nginx controller, and confirm zero nginx.ingress.kubernetes.io annotations remain.


TLS is where the ownership model changes

Routing translates; TLS re-organizes. Under Ingress, each tenant's Ingress carried its own tls: block and cert-manager obliged per object. Under Gateway API, certificates attach to listeners on the operator-owned Gateway, while tenants own only their HTTPRoutes. That split is the one genuine workflow regression to plan for: a tenant who could self-serve TLS by pushing an annotated Ingress can no longer mint a listener. Concretely, three things change:

  1. Cert issuance moves to the Gateway. Wildcard certs for Track A and per-tenant certs for Track B are requested once, against Gateway listeners via certificateRefs, rather than per Ingress. Enable cert-manager's Gateway API support (--enable-gateway-api) and move the HTTP-01 solver from the ingress-nginx solver to the Gateway-native gatewayHTTPRoute solver so challenges route through the new data plane during the migration window.
  2. Challenge traffic must reach the new data plane. During shadow operation, HTTP-01 challenges still resolve to ingress-nginx unless you steer them. Migrate Track A's wildcard (usually DNS-01, no HTTP path involved) first, then flip each Track B host's challenge path as part of its cutover — a host whose cert renews against the old controller the week after cutover is a host that pages you at 3 a.m.
  3. Per-team TLS self-service is still maturing. The experimental ListenerSet (standardized in Gateway API v1.5) is designed to restore per-team listener management on a shared Gateway, and cert-manager has been building toward it — but until that path is GA in both projects, treat TLS as an operator-owned step in the deploy pipeline, with automation rather than self-service doing the per-tenant work.

Five things a clean translation doesn't prove

This is the checklist that makes "successful YAML translation is insufficient proof" operational. Every row is a real warning class from the Ingress2Gateway 1.0 release:

#What translated (or warned)What to verify live
1Timeouts mapped to timeouts.requestFire a request that takes longer than the old TCP timeout but shorter than the new request timeout, and vice versa. TCP-idle and request-duration are different clocks; confirm which one your slowest tenant endpoint actually needs.
2proxy-body-size dropped (implementation defaults apply)POST your largest real payload — container image push, CSV import, whatever your tenants actually upload — against the Gateway default before cutover.
3Regex emitted as (?i) case-insensitive prefixHit the route with wrong-case and over-long paths. If /USERS/123/admin now matches where it didn't, strip the (?i) and trailing .* for an exact match.
4URL normalization untranslatableSend encoded slashes, dot segments, and trailing-dot hosts at both data planes and diff. Normalization behavior is implementation-defined; your tenants' clients depend on the old behavior whether they know it or not.
5configuration-snippet unsupportedFor every snippet, write the behavior as a test first (auth rejects, header present, Lua redirect fires), then reproduce it in the new implementation's policy CRD and watch the test go green.

Pick your GatewayClass on stable ground

One reason to migrate now rather than last year: Gateway API v1.5 (February 2026) is a graduation release, not a feature release — six capabilities promoted from Experimental to Standard, with TCPRoute and UDPRoute following in v1.6. Tenant custom-domain routing (HTTP + TLS) now rests on stable APIs, which is exactly what you want beneath every tenant domain at once.

As for which implementation serves the Gateway, all three production-grade options differ mainly in how they absorb your step-4 leftovers:

  • Envoy Gateway has the most documented zero-downtime migration playbook and first-class traffic-shifting semantics.
  • NGINX Gateway Fabric (F5/NGINX Inc.'s Gateway API product — distinct from the retired community ingress-nginx) keeps NGINX semantics for teams whose snippets and tuning assume NGINX behavior.
  • Traefik advertises itself as the drop-in with the broadest annotation coverage, for teams that want the smallest behavioral delta.

Pick whichever one's policy extensions most naturally express your unsupported-bucket leftovers.

The uncomfortable truth is that the retirement already made this decision for you — the only choice left is whether you migrate on a schedule or under incident pressure. Ingress-nginx will keep proxying traffic while its known vulnerabilities accumulate silently at your edge. Run the inventory this week: it takes an afternoon, it commits you to nothing, and it tells you whether your fleet is a Track-A-canary plus forty clean hosts or a museum of Lua snippets. Either answer is better heard from a census than from a CVE.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources

  • Kubernetes Blog, "Ingress NGINX Retirement: What You Need to Know" (Nov 2025) — kubernetes.io
  • Kubernetes Blog, "Announcing Ingress2Gateway 1.0: Your Path to Gateway API" (Mar 2026) — kubernetes.io
  • Kubernetes Blog, "Gateway API v1.5: Moving features to Stable" (Apr 2026) — kubernetes.io
  • cert-manager, "Ingress-nginx End-of-Life: What cert-manager Supports Today and What's Coming" (Nov 2025) — cert-manager.io
  • CNCF On-Demand, "Zero-Downtime Migration: Ingress NGINX to Envoy Gateway" — cncf.io
  • GKE docs, "Migrate Ingress to Gateway API" — cloud.google.com

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex