Gateway API on kind: The Local Reproduction Recipe for Ingress Bugs You Can't Debug on Bare Metal

A tenant files a ticket: their custom domain returns a TLS error, or their canary route is bleeding traffic to the wrong backend, or their HTTPRoute just never attaches at all. On a cloud-managed cluster, you'd spin up a scratch namespace and poke at it. On a Cluster-API-provisioned Hetzner fleet, the "cluster" is real bare-metal hardware you already promised other tenants wouldn't go down — you can't kubectl delete your way out of a bad experiment.

Kubernetes' January 28, 2026 blog post (by Red Hat's Ricardo Katz) shows a fix for exactly this gap: cloud-provider-kind turns a local kind cluster into a full Gateway API sandbox — a real GatewayClass, a real controller, real HTTPRoute reconciliation — running in Docker containers on your laptop. The stock walkthrough gets you a working Gateway and one HTTPRoute in about five minutes. It does not, on its own, reproduce the three bug classes that actually page a self-hosted PaaS's on-call: a tenant's route silently rejected by a namespace selector, a custom domain that doesn't match your wildcard listener, and a TLS secret a ReferenceGrant never authorized. Below is the stock recipe verbatim, then the extensions that turn it into an actual pre-production repro environment — with the exact kubectl status output each bug produces, so you know what you're looking at before it happens on hardware that costs minutes, not seconds, to fix.

The stock recipe​

This is Kubernetes' own setup, reproduced command-for-command. If you've never run Gateway API locally, start here — it's the baseline every extension below builds on.

kind create cluster

That's a full control plane in a Docker container. Next, cloud-provider-kind — it does two jobs at once: it's a LoadBalancer controller (so Service type LoadBalancer actually gets an IP on a laptop, which kind doesn't do natively) and a Gateway API controller, CRDs included:

VERSION="$(basename "$(curl -s -L -o /dev/null -w '%{url_effective}' \
  https://github.com/kubernetes-sigs/cloud-provider-kind/releases/latest)")"
docker run -d --name cloud-provider-kind --rm --network host \
  -v /var/run/docker.sock:/var/run/docker.sock \
  "registry.k8s.io/cloud-provider-kind/cloud-controller-manager:${VERSION}"

It auto-provisions a GatewayClass named cloud-provider-kind — nothing to create there. Point a Gateway at it:

apiVersion: v1
kind: Namespace
metadata:
  name: gateway-infra

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: gateway-infra
spec:
  gatewayClassName: cloud-provider-kind
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      hostname: "*.exampledomain.example"
      allowedRoutes:
        namespaces:
          from: All

Then an HTTPRoute for one tenant-shaped hostname, and a curl --resolve against whatever IP cloud-provider-kind assigned the Gateway. It works. It's also the last time in this post that allowedRoutes.namespaces.from is set to All — the source post itself flags that as a demo-only shortcut, and it's the first thing to change once you're reproducing multi-tenant behavior instead of a hello-world route.

Bug #1: a tenant's route rejected by the namespace selector​

Bex.co's own Gateway API layer scopes route attachment per tenant namespace rather than leaving it wide open — the exact pattern the stock demo skips. Flip the listener to match that shape:

allowedRoutes:
  namespaces:
    from: Selector
    selector:
      matchLabels:
        gateway-access: "allowed"

Create a tenant namespace without the label, apply an HTTPRoute in it, and check status:

kubectl get httproute tenant-route -n tenant-acme -o jsonpath='{.status.parents[0].conditions}'

[{"type":"Accepted","status":"False","reason":"NotAllowedByListeners", ...}]

That's the ticket from the intro, reproduced on purpose: a tenant's route that never attaches, with no error surfaced anywhere near the tenant's own dashboard — only in HTTPRoute status, which is exactly why this needs a local repro loop instead of a guess against production. Label the namespace (kubectl label namespace tenant-acme gateway-access=allowed) and the same route flips to Accepted: True on the next reconcile, with no other change. Now you've confirmed the fix in seconds, on a cluster you're about to delete anyway.

Bug #2: a wildcard listener that doesn't cover the tenant's real domain​

The stock listener's hostname is *.exampledomain.example — a wildcard. Real tenant custom domains aren't subdomains of your platform's apex; a tenant brings blog.customerdomain.com and CNAMEs it at you. Gateway API requires the HTTPRoute's hostname to intersect with a listener's hostname — a wildcard covers anything.exampledomain.example, but it covers nothing outside that suffix. Apply an HTTPRoute for the tenant's literal domain against the unmodified stock listener:

spec:
  parentRefs:
    - name: gateway
      namespace: gateway-infra
  hostnames:
    - "blog.customerdomain.com"

kubectl get httproute tenant-domain -n tenant-acme -o jsonpath='{.status.parents[0].conditions}'

[{"type":"Accepted","status":"False","reason":"NoMatchingListenerHostname", ...}]

No overlap between listener and route hostname means no attachment — silently, with the route object itself looking perfectly well-formed. The fix in production is a second listener with no hostname field at all (matches any host, and Bex.co's own custom-domain provisioning adds one per tenant CNAME rather than relying on a single catch-all, for the same isolation reason Selector beat All above). Reproducing the failure mode locally first means the first time you see NoMatchingListenerHostname isn't while a real tenant's domain is down.

Bug #3: a TLS secret the Gateway isn't allowed to read​

Add an HTTPS listener terminating TLS from a Secret — generate a throwaway cert with openssl and load it:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout tls.key -out tls.crt -subj "/CN=blog.customerdomain.com"
kubectl create secret tls tenant-cert -n tenant-acme --cert=tls.crt --key=tls.key

listeners:
  - name: https
    port: 443
    protocol: HTTPS
    hostname: "blog.customerdomain.com"
    tls:
      mode: Terminate
      certificateRefs:
        - kind: Secret
          name: tenant-cert
          namespace: tenant-acme

A Gateway's listeners live in gateway-infra; the tenant's cert lives in tenant-acme. That's a cross-namespace reference, and Gateway API requires an explicit ReferenceGrant in the target namespace before it's honored — without one:

kubectl get gateway gateway -n gateway-infra -o jsonpath='{.status.listeners[1].conditions}'

[{"type":"ResolvedRefs","status":"False","reason":"RefNotPermitted", ...}]

The Gateway still reports Programmed: True — the listener itself came up fine — while curl against port 443 hangs or resets, because the one condition that actually explains why is on a different status field than the one most kubectl get gateway scripts check. Add the grant and the same listener resolves cleanly:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-gateway-cert
  namespace: tenant-acme
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: Gateway
      namespace: gateway-infra
  to:
    - group: ""
      kind: Secret

Delete the ReferenceGrant again and swap in a Secret name that doesn't exist at all — the reason flips to InvalidCertificateRef instead of RefNotPermitted. Two different root causes, two different reasons, both invisible to curl and both reproducible without a single Hetzner API call.

A quick-reference table for the three bugs​

Each of these three failure modes looks identical from the outside — a route that "just doesn't work" — but resolves to a different object, a different status field, and a different fix:

None of these four reasons overlap with Programmed, the condition most kubectl get gateway one-liners check first — which is exactly why they're easy to miss without a repro environment that forces you to go read .status.listeners[*].conditions and .status.parents[*].conditions directly.

Wiring it into a pre-merge check, not just a debugging session​

Reproducing a bug after a tenant reports it is still reactive. The same kind + cloud-provider-kind setup runs cheaply enough to sit in CI: spin up the cluster, apply the platform's actual GatewayClass/Gateway/ReferenceGrant manifests (the real ones from the fleet's GitOps repo, not the toy YAML above), apply a representative set of tenant HTTPRoutes and Secrets, and assert on the status conditions before merging a change to any of them. A pull request that flips a Selector back to All, or drops a ReferenceGrant during a refactor, fails a kubectl get httproute -o jsonpath assertion in under a minute of CI time — instead of surfacing as a support ticket from a tenant whose custom domain went dark the moment the change reached the Hetzner fleet. That's the difference between "before it hits production" as an aspiration and as a gate that actually blocks a bad merge.

Why seconds vs. minutes matters more on hardware you own​

Every one of the three bugs above took a kubectl apply and a status check to reproduce, and kind delete cluster && kind create cluster resets the whole environment in under a minute if you want a clean slate. A Cluster-API-provisioned HetznerBareMetalHost node takes minutes to boot, join, and get its CNI/CCM/CSI conditions to a schedulable state — the same order-of-magnitude gap this list has already measured between CAPD's container-backed fake nodes and a real Hetzner provision. Testing a ReferenceGrant fix against a live node pool doesn't just cost those minutes once; it costs them every time the first guess is wrong, on a cluster serving other tenants' traffic while you guess.

The other cost is state, not just time. A wrong allowedRoutes selector or a missing ReferenceGrant tested directly against production doesn't just fail — it fails against a real tenant's real DNS and real TLS termination, for however long the fix takes. The kind-based loop above turns "guess against the fleet" into "confirm locally, then apply once" for the exact bug shapes a multi-tenant Gateway API layer produces, not the single happy-path route the stock demo stops at.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.