Your Load Balancer Finally Tells You Why the Target Is Down: Hetzner's Health-Check Reason Codes, Decoded Into a Runbook

It is 3 a.m., your status page is red, and the load balancer console says exactly one useful thing: a target is unhealthy. Is the app wedged? Did a deploy break the health endpoint? Did someone change a firewall rule?

The console cannot tell you, so you start a second round of probing — curl the target directly, check the deploy log, diff the firewall — while the 502s keep coming.

Since August 17, 2026, Hetzner Cloud load balancers answer that first question themselves. Load Balancer API responses that include the target field now carry targets.health_status.detail, a reason code explaining why a target is failing its health check, and targets.health_status.http_status_code, the actual status the target answered with. Binary up/down became a diagnosis.

This post maps each of the six reason codes to its owner and fix, then wires the codes into Prometheus alerting so the page arrives with its cause attached. The short version is the table below; the rest of the post turns it into a runbook.

What actually changed on August 17​

Before the change, a target's health status was a boolean with a trench coat on: healthy or unhealthy, and nothing else. If you wanted the why, you reproduced the health check yourself — same path, same expected status codes, same expected body — and hoped the failure was still happening when you looked.

The August 17, 2026 changelog entry, "Load Balancer health checks now report why a target is unhealthy," adds the checker's own verdict to every Load Balancer response that returns the target field with a 200 status:

• targets.health_status.detail — set only while a target is unhealthy, one of the six values in the table above.
• targets.health_status.http_status_code — set only together with unexpected_http_status, holding the status code the target actually answered with.

Two properties of that contract matter below. First, detail disappears when the target recovers — absence of a reason means healthy, and any poller you build must treat a vanished detail as a recovery, not stale data. Second, http_status_code is evidence, not a guess: when the code says unexpected_http_status and hands you a 302, you do not need to re-run the probe. That is the entire second round of probing, deleted.

You can see the fields today with the CLI you already have:

hcloud load-balancer describe my-lb -o json \
  | jq '.targets[] | {status: .health_status.status,
      detail: .health_status.detail,
      http_status: .health_status.http_status_code}'

If every target is healthy, detail and http_status come back null — the API telling you there is nothing to explain.

The six codes, decoded​

Hetzner's own descriptions are one line each; the operational content is in what each code rules out. Read the table top to bottom as a network-stack walk: each row assumes the rows above it passed.

layer4_no_connection — the connection could not be established. The app never saw a byte, so stop reading app logs. The usual suspects, in order: a firewall rule change that dropped the LB-to-target path, the service on the target port stopped or never started after a reboot, or the target pointing at a server that no longer exists. Owner: whoever last touched the network path — firewall rules, target registration, the service supervisor.

layer4_timeout — the connection attempt timed out. Distinct from refused: something accepts the idea of a connection and never completes it, or packets die silently in transit. Think overloaded host with a full connection backlog, packet loss on the private network path, or a firewall that drops instead of rejecting. Owner: still the network/host layer, but host load and conntrack tables come before the firewall diff.

layer7_timeout — connected, request sent, answer never arrived. The network path works; the application accepted the request and failed to respond within the check timeout. Classic wedged-worker signature: a deploy mid-restart holding the port open with nobody home, a health endpoint running a query that now outlasts the timeout, an event loop blocked on a downstream call. Owner: the app team, and the deploy log opens first — "it worked ten minutes ago" plus layer7_timeout almost always means something changed in the app or its dependencies.

unexpected_http_status — the target answered with the wrong status. The app is alive; it just disagrees with the check about what "fine" looks like, and http_status_code tells you exactly what it said. A 302 usually means forced-HTTPS or an auth redirect landed in front of the health path. A 404 means the path moved or the app on that port changed. A 500 means the app's own health logic is failing — the one case in this row where the app is genuinely sick rather than misconfigured relative to the check. Owner: app team; redirect/404-class answers are usually check-vs-app drift, 5xx-class answers are real failures.

unexpected_http_content — right status, wrong body. The status matched but the body did not match the check's expected content. The sneakiest row: the endpoint is up, the status is 200, and a naive "is it up?" probe calls it healthy. Typical causes are a framework upgrade that reworded the page the check matches against, a health endpoint rendering an error page with a 200 status, or a check path pointed at content that legitimately changes. Owner: app team — diff the body against the expected content, then decide which side is wrong.

unspecified — the checker does not know. Treat as "unknown," never "probably fine." This code keeps the runbook honest: because the classifier has a bucket for its own ignorance, the other five codes actually mean what they say. Owner: whoever is on call, probing directly as in the old days — with the consolation that every other code now routes itself.

The 502 triage runbook: five scenarios​

The promise of reason codes is that a tenant-facing 502 routes to the right owner without a second probing round. Here is what that looks like per scenario, assuming a standard setup: an HTTPS service behind the LB, an HTTP health check against /health expecting status 200.

Scenario 1: layer7_timeout during a deploy. The deploy restarts workers; a migration runs long; checks time out at L7. Old world: page fires for "target unhealthy," on-call curls the target, sees a hanging connection, guesses deploy-related, opens the deploy log. New world: the alert carries detail="layer7_timeout" — "network fine, app not answering" — and routes straight to the app owner. First action: check for a deploy or migration in flight and decide roll-forward vs. rollback. If none, it is a wedged worker or a suddenly slow dependency — same owner, different tool (app metrics, slow-query log).

Scenario 2: unexpected_http_status with http_status_code: 302 after an app change. Somebody enabled "force HTTPS," and now /health over plain HTTP answers 302 instead of 200. Old world: target flaps unhealthy, on-call reproduces with curl, discovers the redirect, then hunts the change that caused it. New world: the alert hands you the 302. The app is demonstrably alive — it answered, with a redirect — so this is check-vs-app drift, not an app outage. First action: exempt the health path from the redirect or update the check's expected status codes. No 3 a.m. heroics; the service may be serving HTTPS traffic fine while the check argues about HTTP.

Scenario 3: layer4_no_connection after a firewall change. A tightened rule drops the LB-to-target path. Old world: identical console symptom to scenario 1 ("target unhealthy"), diverging only after manual probing. That is the point of the comparison — scenarios 1 and 3 looked the same before August 17 and route to different teams now. The app never saw the request, so the app team has nothing to investigate. First action: diff recent firewall changes and verify a listener on the target port. With declarative firewall management, look at the last reconciliation, not the last deploy.

Scenario 4: unexpected_http_content after a framework upgrade. The upgrade reworded the page the check matches on; status still 200, body no longer matches. Nothing is actually broken — but the target reads unhealthy and the LB drains it, creating a capacity incident out of a cosmetic diff. That is why the content code gets its own row: the failure it warns about is the load balancer removing healthy capacity. First action: diff the body against the expected content, then decide which side is canonical. Long-term: point checks at a dedicated /health endpoint with a stable, minimal body instead of matching marketing copy.

Scenario 5: unspecified, the honest fallback. The checker does not know. Route it like the old world — on-call probes directly — with one free improvement: unspecified arriving without any specific code firing fleet-wide means the failure is genuinely odd, not merely common-but-unclassified. Treat it as unknown-severity: acknowledge fast and probe. If it pages twice in a week, that recurrence is the signal — something fails in a way the taxonomy cannot see, and that deserves investigation, not a shrug.

Wiring it into Prometheus so the page carries the cause​

Reason codes only shorten incidents if they reach the on-call human. The metrics API and community LB exporters cover traffic and service-state metrics, not per-target health verdicts — so the reason codes need a small poller of their own. The pattern is deliberately boring: poll the Cloud API, expose a gauge, alert on labels.

Auth and scope. Create a dedicated read-only API token for the project holding the load balancers. The poller needs exactly one permission — reading LB state — so a leaked token cannot change anything. Pass it as an environment variable, never a flag or a committed file.

Poll cadence. Poll no faster than your health-check interval. With 15-second checks and 3 retries, a 60-second poll sees every verdict that survives long enough to matter and never pages on a self-healed flap. Faster polling cannot detect real failures sooner — the checker's interval is the floor — it just burns rate limit.

Metric shape and the staleness rule. Expose one gauge per (load balancer, target, service), with detail and http_status_code as labels:

hetzner_lb_target_unhealthy{lb="prod-lb",target="app-03",detail="layer7_timeout",http_status_code=""} 1

The critical behavior comes straight from the API contract: detail exists only while the target is unhealthy. So the poller must reset — when a target recovers, drop the series or set it to 0 with empty reason labels. A series that keeps its last detail after recovery pages the next on-call for an incident that ended.

Simplest correct approach: every poll sets 1 for currently-unhealthy targets with current labels and 0 (or deletes) for everything else. Never carry a reason label across a healthy poll.

Cardinality. The label set is naturally bounded: six detail values, and http_status_code set on only one of them. Per-target series scale with your fleet like the per-target series you already have. The one trap is churning target IPs (label-selector targets on autoscaled pools) in a label without cleanup — pair the poller with series cleanup for removed targets, or label by stable server name.

The alert rule. One rule routes by label, which is the whole payoff:

- alert: LoadBalancerTargetUnhealthyWithCause
  expr: hetzner_lb_target_unhealthy == 1
  for: 2m
  labels:
    severity: page
  annotations:
    summary: "Target {{ $labels.target }} on {{ $labels.lb }} unhealthy: {{ $labels.detail }}"
    runbook: "detail=layer4_* → network/firewall owner; detail=layer7_timeout|unexpected_* → app owner with deploy log; detail=unspecified → probe directly"

A minimal poller is a few dozen lines of Python against GET /v1/load_balancers with the read-only token, translating each target's health_status into the gauge above. Run it next to your existing exporter, and the 3 a.m. page changes from "something is unhealthy, good luck" to "app-03 is timing out at L7, here is the runbook line."

Limits and gotchas​

The code is a verdict, not a log. You get the current reason, not its history. If a target flaps between layer7_timeout and healthy, the API shows whichever is true at poll time — reconstruct timelines from your alert history, which now carries the reason.

Check configuration is half the diagnosis. unexpected_http_status and unexpected_http_content compare the answer against your configured expectations — status codes, body, path, host header. A wrong expectation produces a confident, precise, wrong-looking verdict. When one fires, read the check config alongside the verdict before blaming the app. Scenario 2's 302 is the canonical example.

Timeouts have two authors. layer7_timeout measures the checker's patience — your configured timeout. If it is shorter than your app's honest worst case (cold JVM, first-query plan compilation), the code correctly reports a late answer, and the fix is a realistic timeout plus a faster app, not one of them.

Retries delay the verdict. The LB marks a target unhealthy only after the configured retries fail, so the code reflects a sustained verdict. Good for alert quality — but it appears retries-times-interval after the failure starts, so size your alert for: accordingly instead of stacking a second delay.

TCP checks cannot produce HTTP verdicts. The three HTTP-specific codes exist only for HTTP health checks. A TCP-checked service serving broken HTTP reads layer4-healthy while users get errors — the codes upgrade L7 visibility only where you configured L7 checks.

The one-line version​

Binary health status made every incident start with "which layer is broken." Six reason codes move that question from incident time to configuration time: map each code to an owner once, teach the mapping to your alerting, and the next 3 a.m. page arrives pre-routed. The API contract is small enough to hold in your head — detail only while unhealthy, http_status_code only with unexpected_http_status — and the poller is small enough to write in an afternoon. That is a good trade for deleting the least fun twenty minutes of every load-balancer incident.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.