Interview Kubernetes vs Production Kubernetes: What a Viral Essay Gets Right About Running Clusters on Owned Machines

The most common question in a Kubernetes technical interview is "what is a Pod?" The second most common is "what's the difference between a Deployment and a StatefulSet?" Both are valid. Both are almost irrelevant to 80% of the problems that actually break a cluster in production.

That opening — from Juan Torchia's essay What Job Interviews Taught Me About Kubernetes, which climbed Hacker News' front page in mid-June 2026 — is the sharpest one-paragraph diagnosis of the Kubernetes skills gap I've read this year. Torchia's thesis: interviews don't measure what you can use, they measure what you can define fast under pressure. And that creates a deeply distorted map of Kubernetes — packed with API objects you'll barely touch, with almost no room for the operational decisions that actually matter.

He's right. And the distortion hurts most exactly where this blog lives: teams running Kubernetes on machines they own, where there is no managed control plane to absorb the mistakes. So let's take the essay seriously as a spec — map each of its claims onto a self-managed fleet, and extract the hiring rubric hiding inside it. Here is the whole argument up front.

The map, corrected for owned machines​

Torchia contrasts the two lists every Kubernetes operator carries in their head. The interview list — Pod, Service, ConfigMap/Secret, Ingress — is the set of objects easiest to define in one sentence. The postmortem list — PodDisruptionBudget, ResourceQuota/LimitRange, HPA with custom metrics, affinity and tolerations — is what shows up in real incident reports. The distance between those two lists, he writes, is the map he was missing.

On a managed cluster, that distance costs you slow incident response. On owned hardware, it costs you outages, because the failure modes land harder without a cloud provider's guardrails. Mapped row by row:

The rest of this post works through the three rows that page self-hosters most often — PDBs, memory limits, DNS — and then turns the essay into an interview rubric you can use on Monday.

PodDisruptionBudgets: the drain that takes down prod​

Node maintenance happens. Kubernetes drains the node. All three of your pods were on that node. Your service is now completely down — and from the cluster's perspective, nothing went wrong. Every eviction was voluntary, policy-compliant, and logged.

This is the single most common "we had no PDB" story in production writing this year, and the fix is eleven lines of YAML:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-api

For an app with three replicas and minAvailable: 2, Kubernetes may evict one pod during maintenance but must not voluntarily evict two at once. Node drains, cluster upgrades, and autoscaler scale-downs all respect it. Involuntary disruptions — a kernel panic, a dead disk — do not, which is exactly the point: a PDB is a budget for the disruptions you schedule.

Here is where owned machines change the stakes. On EKS or GKE, managed node groups rotate instances with surge settings and health checks that partially cover for a missing PDB. On a Cluster-API-managed fleet on Hetzner, you are the rotation mechanism: every MachineDeployment rollout, every kernel upgrade, every Talos or Flatcar update drains nodes. No PDB means every maintenance window is a game of "which single-node service falls over this time." The PDB isn't best practice on owned hardware. It is the maintenance window.

And the failure mode cuts both ways, which is why PDBs deserve understanding rather than cargo-culting. As Neel Shah wrote in June 2026, PDB misuse causes a disproportionate share of production incidents — and the second most common mode is the over-aggressive PDB that blocks drains entirely. The classic: minAvailable: 1 on a Deployment with replicas: 1, where the API server refuses to evict the old pod during a rolling update and the rollout stalls forever.

The rule that survives both directions: maxUnavailable on multi-replica services (it degrades gracefully at small replica counts), minAvailable only once you have enough replicas that the arithmetic can't wedge, and a PDB on every workload that serves traffic — including the single-replica internal tools, where the honest budget is "this will be briefly down" encoded as maxUnavailable: 1.

If your fleet has exactly one operational gap to close this quarter, audit PDB coverage. kubectl get pdb -A takes five seconds and tells you which services your next upgrade will take down.

Memory limits: the OOMKiller doesn't negotiate​

Torchia's formulation is admirably blunt: if you don't configure requests, the scheduler is flying blind; if you set limits too tight, the OOMKiller will murder your process without warning. The owned-machine corollary: on fixed hardware, somebody else's misconfigured limit becomes your eviction.

Two rules have hardened into consensus across this year's production writing, and both contradict what interviews reward:

Memory: set requests and limits, and set them equal. Memory doesn't burst harmlessly the way CPU does. When requests are lower than limits, you invite the scheduler to overcommit RAM the node may not actually have when your pod needs it — and the cgroup OOM killer resolves the disagreement by terminating a process with exit code 137. Equal requests and limits put the pod in the Guaranteed QoS class and make scheduling honest: if the node can't fit the pod's real footprint, the pod stays Pending instead of landing and dying later. On a cloud node pool with headroom, overcommit is a bet that usually pays. On a right-sized Hetzner box, it's a bet against yourself.

CPU: set requests honestly, and think twice before setting limits at all. CPU and memory fail asymmetrically — CPU throttles, memory kills. A container that exceeds its CPU limit doesn't die; it gets artificially slowed, which manifests as p99 latency that looks exactly like an application bug. As Daniel Valev put it in June 2026: memory limits contain blast radius (the leaking container dies alone in its own cgroup and the node lives), while CPU limits manufacture a performance problem and then bill you in debugging hours. The exception is the noisy-neighbor guarantee for untrusted multi-tenant workloads — precisely the case a PaaS should handle with QoS classes and the new tiered Memory QoS controls in Kubernetes 1.36 rather than blunt per-container CPU ceilings.

One honest caveat, borrowed from the essay itself: Torchia is explicit that his post is pattern analysis, not production measurement, and the same applies here. Whether equal-requests-limits or a looser ratio is right for your workload mix is answerable only with your own memory telemetry — VPA recommendations, container_memory_working_set_bytes history, and a record of which pods actually OOM. The interview answer is "requests are reservations, limits are ceilings." The production skill is reading the graphs that tell you what numbers to put in those fields.

DNS: the five-second timeout nobody interviews for​

No interviewer has ever asked a candidate about ndots:5. No on-call rotation on Kubernetes has ever avoided it. The most reported DNS symptom in the ecosystem is lookups that usually work but occasionally take exactly five seconds — the glibc retry interval — or time out entirely. On owned machines, where there is no cloud DNS floor beneath CoreDNS, this failure mode is yours end to end.

The mechanism is worth understanding once, because it explains three separate incidents. Every pod's resolv.conf inherits options ndots:5 from the kubelet, plus a search list (default.svc.cluster.local, svc.cluster.local, cluster.local, …). Any name with fewer than five dots — which is to say, every external hostname your app resolves, like api.stripe.com — gets tried against each search domain first, generating a storm of internal queries (and NXDOMAIN responses) before the real lookup happens.

Under load, two things break: CoreDNS pods CPU-pin serving NXDOMAINs they never needed to see, and the NAT conntrack table on the node races UDP replies for the burst of parallel queries, dropping some — which surfaces, five seconds later, as a glibc retry. The app sees intermittent 5s stalls. The dashboards show nothing, because 95% of queries succeed fast.

The mitigations are all cheap, which is what makes this a skills problem rather than a tooling problem:

• Use fully-qualified names with a trailing dot (api.stripe.com.) in application config for hot-path external calls — this skips search-path expansion entirely.
• Lower ndots (to 1 or 2) via dnsConfig on pods that mostly call external hosts.
• Run NodeLocal DNSCache so the per-node cache absorbs the search-path storm instead of every query traversing the CNI to a CoreDNS pod.
• Scale CoreDNS and enable negative caching (cache 30 with denial caching) so the unavoidable NXDOMAIN load stops pinning the server pods.

Notice the shape of this knowledge: nothing about it is an API object you can define in one sentence. It spans libc resolver behavior, kernel conntrack, Corefile plugin semantics, and DaemonSet capacity planning. That spanning — not any single fact — is what "production Kubernetes" means, and it's exactly what definition-style interviews can't see.

What to hire for: steal the essay's checklist, fix the interview​

Torchia's most actionable section is his "real priority checklist": five kubectl commands he claims cover 70% of initial day-one diagnostics — rollout status, sorted events, container resources as JSONPath, HPA state, and non-Running pods. It's a good triage list. But the deeper gift of the essay is that it accidentally specifies a better interview: stop asking candidates to define objects, and start asking them to operate.

The industry is already moving that way. A 2026 survey of production incident runbooks notes that senior panels have largely abandoned "what is a Pod?" in favor of simulated fire drills — silent cgroup OOM kills, CoreDNS latency under surge, rolling updates that wedge. If you're hiring for a self-managed fleet, here are five scenario questions in that spirit, each mapped to a section above:

1. "You're draining a node for a kernel upgrade and a three-replica service goes fully dark. Walk me through it." You're listening for: voluntary vs involuntary disruption, PDBs, kubectl get pdb, and what they'd check before the next drain. Bonus if they mention the over-strict-PDB stall in reverse.
2. "A pod restarts every few hours with exit code 137, but its memory graph never touches the limit. What do you check?" You're listening for: working set vs RSS vs cache, requests/limits ratio and QoS class, node pressure eviction vs cgroup OOM, VPA data. Bonus if they ask which container in the pod died.
3. "External API calls from pods intermittently stall for exactly five seconds. Dashboards are green. Where do you look?" You're listening for: ndots search expansion, CoreDNS CPU, conntrack drops, NodeLocal DNS. Anyone who says "check the app's HTTP client timeouts first" has been burned by this before — that's a good sign, not a dodge.
4. "Liveness and readiness probes: give me a failure caused by confusing them." You're listening for: liveness killing a slow-starting container into a CrashLoop (needs a startup probe or wider grace), readiness missing so traffic hits a cold JVM. This is Torchia's "most silent error," and it has no definition-shaped answer.
5. "When would you choose Recreate over RollingUpdate?" You're listening for: non-backwards-compatible migrations, single-writer state, the cost (brief downtime) stated plainly rather than waved away. Bonus if they ask about the PDB interaction.

None of these questions has a one-sentence answer, which is precisely the filter. A candidate who can define twelve API objects but can't drain a node without an outage is the essay's warning made flesh. A candidate who reasons through even three of these scenarios can keep your fleet alive — and on owned machines, where every layer is yours, that is the whole job.

Torchia also includes a decision matrix for when Kubernetes itself is overkill, and he's right that a handful of stateless services on Render or Fly.io shouldn't be a cluster. But that matrix has a row he underplays: current platform: on-prem or cloud without PaaS. If you already own the machines — for cost, for data gravity, for independence from any single vendor's price list — then Kubernetes isn't the overkill choice. It's the control plane that turns a pile of servers into a platform, and production fluency in it is the price of admission you pay once instead of the managed PaaS margin you pay forever.

Juan Torchia's essay is What Job Interviews Taught Me About Kubernetes — worth reading in full, including its honest caveats about what you can't conclude without production data. And if the "you already own the machines" row of the matrix is your situation: Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.