ExternalDNS: The Controller That Turns Every Ingress Into a DNS Record (and What Breaks at Fleet Scale)

Every self-hosted PaaS has a moment where someone types my-app.onbex.co into a browser and expects it to resolve. Not after a ticket. Not after someone logs into Cloudflare. Immediately — the instant the Kubernetes object that asked for that hostname is created. The controller responsible is ExternalDNS, a Kubernetes SIGs project that has been quietly turning annotations into A and CNAME records since 2017, and it is the reason a git push can end with a working HTTPS URL without a human touching DNS.

The idea is deceptively simple — annotate a Service or Ingress with a hostname, and the record appears. The implementation at fleet scale is not.

Here is the full reconciliation loop, the ownership model that prevents two clusters from fighting over the same zone, and the rate-limit math that decides whether a 500-app rollout takes 30 seconds or 30 minutes.

The reconciliation loop in one table​

ExternalDNS is a standard Kubernetes controller: it watches objects, computes desired state, diffs it against actual state, and applies the delta. The loop runs on an interval (default --interval=1m), not on every watch event, which is the first hint that this is a batch reconciler, not an event-driven operator.

The loop is idempotent. If the provider already matches desired state, nothing happens. If someone deletes a record by hand, the next interval recreates it. If a Service is deleted, its record is removed — but only if the TXT registry says this ExternalDNS instance owned it.

That last qualifier is the entire fleet-scale story.

Sources: what ExternalDNS watches and how it decides​

By default ExternalDNS watches service and ingress. You select sources explicitly:

args:
  - --source=service
  - --source=ingress
  - --source=gateway
  - --source=crd

Each source extracts endpoints differently:

• Service — reads external-dns.alpha.kubernetes.io/hostname annotation for LoadBalancer and ClusterIP (with --service-type-filter). The target is the Service's status.loadBalancer.ingress IP or hostname. NodePort is not supported.
• Ingress — reads spec.rules[].host and the same hostname annotation. Target is the Ingress controller's load-balancer address or the annotation's target value.
• Gateway API — watches Gateway objects (installed as CRDs). Hostnames come from spec.listeners[].hostname.
• CRD — watches DNSEndpoint custom resources directly, for cases where no networking object exists.
• Annotation filter — --annotation-filter=external-dns.alpha.kubernetes.io/internal!=true lets you run two ExternalDNS instances against the same cluster (e.g., internal vs external zones) without overlap.

The hostname annotation is the universal override. Even when a source derives hostnames from spec fields, the annotation always wins:

apiVersion: v1
kind: Service
metadata:
  name: my-app
  annotations:
    external-dns.alpha.kubernetes.io/hostname: my-app.onbex.co
    external-dns.alpha.kubernetes.io/ttl: "60"
spec:
  type: LoadBalancer

At fleet scale, the annotation is how a PaaS platform assigns <tenant>.onbex.co — the platform controller stamps the hostname onto the Service or Ingress it creates for each tenant app, and ExternalDNS does the rest.

The TXT registry: ownership without a database​

The hardest problem in multi-cluster DNS is not creating records — it is not deleting records you do not own. Two clusters sharing onbex.co must not clobber each other's entries, and neither should delete a record a human created by hand.

ExternalDNS solves this with TXT records co-located with every managed record:

my-app.onbex.co.  60  IN  A      203.0.113.42
my-app.onbex.co.  60  IN  TXT    "heritage=external-dns,external-dns/owner=my-cluster,external-dns/resource=service/default/my-app"

The TXT record encodes three things: that ExternalDNS manages the sibling record (heritage), which instance owns it (owner), and which Kubernetes object produced it (resource). On the next interval, the controller lists all TXT records in the zone, builds an ownership map, and only touches records where owner matches its own --txt-owner-id.

args:
  - --registry=txt
  - --txt-owner-id=my-cluster-prod-eu
  - --txt-prefix=external-dns-

Three details that bite at fleet scale:

1. Every cluster needs a unique txtOwnerId. If two clusters share an ID, the second one will happily delete the first one's records and recreate them pointing at itself. The fix is to derive the ID from the cluster name and make it immutable — changing it orphans every record the old ID owned.

2. CNAME records need a prefix. A TXT record cannot share a name with a CNAME (DNS restriction). If you manage CNAMEs, set --txt-prefix so the TXT lives at external-dns-my-app.onbex.co instead of my-app.onbex.co. Changing the prefix later orphans records the same way changing the owner ID does.

3. Domain filters scope the blast radius. --domain-filter=onbex.co restricts which records the controller considers; --exclude-domains=internal.onbex.co carves out exceptions. Without a filter, a misconfigured controller can enumerate and attempt to manage every record in the zone — which is also how you discover rate limits the hard way.

Provider rate limits: the math that decides rollout speed​

Every DNS provider throttles API calls. ExternalDNS batches changes per interval, but a fleet-wide event — a rolling upgrade that gives 200 Machines new IPs, or a zone migration — can generate hundreds of record changes in a single interval.

The practical consequence: if 500 tenant apps each need a DNS update and the provider allows 5 writes per second, the full reconciliation takes at minimum 100 seconds of API time — spread across intervals if the controller batches conservatively. During that window, some tenants resolve to the old IP and some to the new one. There is no shortcut; the provider's rate limit is the bottleneck, not the controller.

Mitigations that actually help:

• Increase --interval for large fleets — polling faster than the provider can absorb just generates throttled requests that retry anyway.
• Raise --cloudflare-dns-records-per-page (or equivalent) to reduce list-call volume per interval.
• Use --events cautiously — it watches endpoint changes and triggers extra reconciliations, which increases provider API pressure.
• Separate zones per fleet segment — eu.onbex.co and us.onbex.co as distinct zones means each ExternalDNS instance throttles against its own zone, not a shared one.
• Test with --dry-run — logs the plan without calling the provider, so you can measure how many changes a rollout would generate before it hits the API.

Fleet-scale pitfalls: what breaks when you run one zone across many clusters​

Running ExternalDNS on a single cluster is straightforward. Running it across a Cluster API fleet sharing one parent zone is where the operational subtleties live.

Owner collision. The most common incident: a new cluster is provisioned with a copy-pasted ExternalDNS manifest that reuses another cluster's --txt-owner-id. Both controllers now believe they own the same records. The symptom is flapping DNS — records alternating between two IPs on successive intervals — and the fix is always to assign unique owner IDs before the first interval runs.

Orphaned records on cluster deletion. When a workload cluster is decommissioned, its ExternalDNS instance stops running, but its TXT and A records remain. No other instance will delete them (wrong owner), and no human may notice until the zone accumulates hundreds of stale entries. Fleet operators handle this with a decommissioning job that runs ExternalDNS one final time with --policy=sync (which deletes orphaned records) or with explicit DNSEndpoint cleanup.

Policy matters. --policy=upsert-only creates and updates records but never deletes. --policy=sync (the default) deletes records whose source objects no longer exist. At fleet scale, upsert-only is safer during migrations — you avoid a misconfigured controller deleting records it should not — but it leaks stale records. Most fleets run sync in steady state and switch to upsert-only during controlled transitions.

Webhook provider as escape hatch. When the built-in provider list does not include your DNS backend, the webhook provider lets you run any DNS API behind a small HTTP server that implements the ExternalDNS webhook contract. This is how teams front PowerDNS, Porkbun, or internal DNS systems without forking ExternalDNS itself. The tradeoff is operational: the webhook is another service to run, monitor, and keep within rate limits.

Why not hand-roll the reconciliation loop​

It is tempting to write a bespoke controller — watch Ingress objects, call the Cloudflare API, done in 200 lines. Teams that do this rediscover, one incident at a time, why ExternalDNS exists:

• Ownership tracking — without TXT records, every hand-rolled controller must implement its own "who owns this record" logic, or risk the same flapping described above.
• Provider abstraction — switching from Cloudflare to Route 53 (or supporting both for different zones) means rewriting the provider layer. ExternalDNS already supports 30+ providers plus the webhook contract.
• Plan diffing — correctly computing the minimal set of creates, updates, and deletes across hundreds of records, handling CNAME vs A vs TXT co-location rules, is more subtle than it appears.
• Rate-limit handling — retry with backoff on 429, batch sizing, pagination for large zones — all already implemented and battle-tested.

The bespoke controller saves no meaningful complexity and acquires all the failure modes. ExternalDNS is the boring, correct choice — which is exactly why it powers the <name>.onbex.co pattern on every fleet that does it well.

Wiring it into a self-hosted PaaS​

On a Bex-style fleet — Cluster API provisioning Hetzner Machines, Cilium as CNI, Gateway API for ingress — the wiring is:

1. One ExternalDNS per workload cluster, with a unique --txt-owner-id derived from the cluster name, --domain-filter scoped to the tenant subdomain zone, and --registry=txt.
2. Platform controller stamps external-dns.alpha.kubernetes.io/hostname: <tenant>.onbex.co onto the Gateway or Service it creates per tenant.
3. ExternalDNS reconciles the record to the cluster's ingress load-balancer IP. When the Machine is replaced and the IP changes, the next interval updates the record — no human intervention.
4. Cert-manager (or equivalent) handles TLS for the same hostname via DNS-01 or HTTP-01, independently of ExternalDNS — the two controllers coordinate only through the DNS record itself.

The result: git push → build → deploy → Gateway with hostname annotation → ExternalDNS creates my-app.onbex.co → cert-manager provisions TLS → working HTTPS URL. Every step is a controller reconciliation loop; no step requires a human or a provider dashboard.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.