Railway Finally Shipped DNS Logs: What You Can Debug From a Dashboard vs What You Get From CoreDNS on Day One

On July 24, 2026, Railway's changelog carried a line that would have been unremarkable if it had shipped three years earlier: DNS logs. Tenants can now see the queries their services make inside Railway's private network, with response codes and targets visible in the dashboard instead of ending as a support ticket that says "my service can't reach my other service."

The feature matters not because DNS is exotic but because it is the most common invisible failure in private networking. A connection that times out looks like a network problem, a firewall problem, or an application bug. Roughly a third of the time it is a name that did not resolve the way the author assumed. Until now, Railway tenants debugged that by guessing. The rest of this post tours the three ways that guesswork fails, shows what the new dashboard surface actually reveals, and shows what the same three failures look like when the resolver is CoreDNS on a cluster you own — where the logs, the metrics, and the packet capture have been available on day one.

These three account for the majority of "works locally, fails on the platform" DNS tickets. Keep the table handy — the next two sections debug each row twice.

What Railway actually shipped (and what is still behind the vendor's timeline)​

Railway's July 24, 2026 changelog shipped DNS logs alongside a round of log and observability improvements that had been accumulating since June. The surface, from the changelog and the dashboard:

• Per-service query log: each DNS query the service issues inside the private network is recorded with the queried name, query type, response code (NOERROR, NXDOMAIN, SERVFAIL), and the resolved target or failure reason.
• Dashboard location: the logs live alongside the existing service logs, so the workflow is "tail the DNS log while reproducing the failing fetch."
• Private-network scope: the logs cover private-network name resolution — the exact surface where api.railway.internal (or the equivalent Railway private DNS name) either resolves to a service or does not.

What the surface does not include is as telling as what it does:

• No access to the resolver configuration — you cannot read resolv.conf, adjust ndots, or edit the search list.
• No packet capture of the wire — you cannot tcpdump the resolver to see whether a query left the box at all.
• No TTL or cache controls — a stale record is flushed by restarting the service, not by issuing a cache invalidation.

None of this is a criticism of the feature. It is the structural shape of every observability surface on a hosted PaaS: it arrives as a vendor-prioritized feature, on the vendor's timeline, scoped to what the vendor chooses to expose. Railway's private networking has existed for years; DNS logs arrived when they were prioritized relative to every other roadmap item. Before that date, the only DNS debugging surface was inference — "the fetch failed, so the name must not have resolved" — plus a support ticket. After that date, there is a dashboard panel. Both are better than nothing, and both are exactly one surface.

The pattern is worth naming because Railway is not an outlier. Fly.io's private networking, Render's private services, and Vercel's build-time DNS all expose name resolution as an opaque success or failure until the vendor decides otherwise. The capability to instrument the resolver exists on day one inside every one of those platforms — the vendor just has to productize it.

Debugging each failure on Railway (rented surface)​

With DNS logs available, each of the three failures moves from "guess and restart" to "read the log and act," but the fix is still constrained to what the tenant can change.

1. ndots trauma on Railway​

What you see in DNS logs: a single fetch("https://api.stripe.com/v1/charges") produces three or four log lines — api.stripe.com.staging.svc.cluster.local → NXDOMAIN, api.stripe.com.svc.cluster.local → NXDOMAIN, then api.stripe.com → NOERROR — each costing a 5-30 ms round-trip. The external call that should have taken one lookup takes three or four.

What you must infer: the dashboard does not label these as "search expansions." You recognize the pattern by the repeated NXDOMAIN lines sharing a prefix of the target name.

Workaround: append a trailing dot to the external name wherever the HTTP client allows it — api.stripe.com. — which signals "this is absolute, do not search." Where the client does not allow it, you cannot fix the search penalty from the tenant side; you can only observe it.

2. Search-domain expansion to the wrong environment on Railway​

What you see: your production service queries api expecting api.production but the log shows api → api.staging.svc.cluster.local → NOERROR — a successful resolution to the wrong IP. The request succeeds at the DNS layer and fails at the application layer (wrong data, auth rejection, or a subtle logic error).

What you must infer: DNS logs show the expansion path, but they do not show which private service api.staging actually is without cross-referencing the service list. A bare api query is ambiguous by design.

Workaround: always use the fully-qualified private name (api.production.railway.internal or the Railway-equivalent) instead of the short name. This is a convention fix — nothing enforces it.

3. Stale service records on Railway​

What you see: after redeploying or scaling worker from two to four replicas, the log for worker.railway.internal continues showing the old IP for a window after the deploy.

Workaround: restart the querying service to flush its local cache. There is no tenant-visible TTL knob or cache-invalidation API. If the platform's DNS TTL is longer than your deploy cadence, restarts become part of the deploy ritual.

Debugging each failure on your own Kubernetes (owned surface)​

On a cluster you run, the resolver is not an opaque vendor service. It is CoreDNS, a Deployment in kube-system whose ConfigMap, logs, metrics, and network namespace you control. The same three failures are debugged with primitives that have existed since CoreDNS became the default in Kubernetes 1.11.

The resolver you actually own​

Every pod's /etc/resolv.conf points at the cluster DNS Service IP:

nameserver 10.32.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Three things to notice: the search list mirrors Railway's private-network search, the ndots:5 default is identical in effect, and all three are per-pod overrides you can set via dnsConfig — no vendor timeline required.

CoreDNS log plugin: every query, on demand​

Enable query logging in the CoreDNS Corefile (the ConfigMap coredns in kube-system):

.:53 {
    log
    errors
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
}

Adding log writes every query to stdout in the common log format:

10.244.1.23:34567 - 12345 "A IN api.stripe.com.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 146 0.000312s
10.244.1.23:34567 - 12345 "A IN api.stripe.com.svc.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 146 0.000521s
10.244.1.23:34567 - 12345 "A IN api.stripe.com. udp 47 false 512" NOERROR qr,rd,ra 118 0.004231s

To isolate failures, scope the plugin:

log . {
    class denial error
}

Now only NXDOMAIN/NODATA and SERVFAIL/REFUSED lines are emitted — exactly the DNS logs surface, but filtered at the source.

Read them live:

kubectl -n kube-system logs -l k8s-app=kube-dns -f --tail=50
kubectl -n kube-system logs -l k8s-app=kube-dns --since=5m | grep NXDOMAIN

Prometheus metrics: the shape of the failure over time​

CoreDNS exposes coredns_dns_requests_total and related counters on :9153:

# NXDOMAIN rate by zone — spikes when search expansion is misfiring
sum(rate(coredns_dns_requests_total{rcode="NXDOMAIN"}[5m])) by (zone)
 
# Cache hit rate — drops when TTLs are being ignored or negative caching is off
sum(rate(coredns_cache_hits_total[5m])) / sum(rate(coredns_dns_requests_total[5m]))
 
# Forward latency — the cost of ndots-driven upstream queries
histogram_quantile(0.99, rate(coredns_forward_request_duration_seconds_bucket[5m]))

Where Railway's dashboard shows the last N queries, CoreDNS metrics show the trend. A deploy that introduces a short-name reference to api lights up the NXDOMAIN counter in staging.svc.cluster.local immediately — no need to reproduce the bug interactively.

tcpdump: when the log says nothing​

The log tells you what CoreDNS received; tcpdump tells you whether the query left the pod at all:

# Capture DNS on the CoreDNS pod itself
kubectl -n kube-system debug -it $(kubectl -n kube-system get pod -l k8s-app=kube-dns -o name | head -1) \
  --image=nicolaka/netshoot -- tcpdump -ni any port 53 -v
 
# Or capture inside the failing pod
kubectl debug -it <pod> --image=nicolaka/netshoot -- tcpdump -ni eth0 port 53 -A

Two cases where this is the only answer: a dnsPolicy: Default pod that bypasses CoreDNS entirely and talks to the node's upstream resolver, and a network policy that silently drops UDP port 53.

Fixing each failure where you control the config​

ndots trauma — set ndots per workload:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
      - name: edns0
  dnsPolicy: ClusterFirst

With ndots:2, only names with fewer than two dots use the search list. api.stripe.com (one dot → wait, two labels? Actually one dot means one separator — but ndots counts dots, so api.stripe.com has two dots and at ndots:2 it is treated as absolute). The upstream query goes out first, no NXDOMAIN detour. For workloads that make many external calls, consider NodeLocal DNSCache, a DaemonSet that caches on each node and eliminates the ndots penalty for repeated lookups.

Search-domain expansion — be explicit or constrain the search list:

spec:
  dnsConfig:
    searches:
      - production.svc.cluster.local
      - svc.cluster.local
      - cluster.local

Better still, use fully-qualified names with a trailing dot (api.production.svc.cluster.local.) in code. The dot costs nothing and makes the query unambiguous regardless of search configuration.

Stale records — tune the cache and observe TTLs:

cache 5  # seconds, instead of the default 30

Or, for a single service, lower the TTL on the backing Endpoints by adjusting the kubernetes plugin's TTL. Then verify with dig:

kubectl run -it --rm debug --image=infoblox/dnstools -- dig worker.production.svc.cluster.local +nocmd +noall +answer
# worker.production.svc.cluster.local. 5 IN A 10.244.2.18

The 5 is the TTL. When it says 5, you know a redeploy propagates in five seconds, not thirty.

Rented vs owned: the structural difference​

The bottom row is the point. Every row above it is a feature Railway could expose and eventually may. The question for a team choosing where to run production is not whether the hosted PaaS will eventually expose enough of the resolver — it is how many debugging sessions will end at "I can't see that yet" before it does.

On a cluster you own, the resolver is not a feature. It is a Deployment you can read, a ConfigMap you can edit, and a network namespace you can capture. The debugging surface is not a changelog entry. It is an API.

What to do on Monday​

If you run on Railway, the DNS logs panel is the new first place to look when a private-network fetch fails. Tail it while reproducing the failure, scan for the repeated-NXDOMAIN ndots pattern, and adopt two habits that make the log useful: use fully-qualified private names (api.production.railway.internal, not api) and add trailing dots to external names where the client allows it. When a stale record outlives a deploy, restart the caller.

If you run your own fleet under Cluster API or any Kubernetes, ship three things this week if you have not already: log { class denial error } in the CoreDNS Corefile so failures are visible without drowning in success, a dashboard on coredns_dns_requests_total{rcode="NXDOMAIN"} so search-expansion regressions page before users notice, and ndots:2 (or NodeLocal DNSCache) for workloads that call external APIs on every request. The next time a service cannot reach another service, the answer will be in the log you already ship, the metric you already alert on, or the capture you can take in one command.

Self-hosting does not make DNS simpler. It makes DNS debuggable — on your timeline, not a vendor's changelog. Bex is the open-source, self-hosted PaaS that runs on your Kubernetes — same deploy-from-git workflow, but the resolver, the logs, and the packet capture stay on machines you own. The DNS debugging surface is not a feature you wait for. It is the platform.