Dora NodaAt 3am last Tuesday, somebody scaled your production API to zero — and every dashboard is green. No failed deploy, no error spike, no alert fired. The rollout history shows a scale-down to zero replicas, requested through the API with a valid token, twelve minutes after a routine dependency bump. The token belongs to your deploy agent.
Which session, what was it trying to do, did a human approve it, and what did it cost in tokens before it acted? If you cannot answer all five in under a minute, your agents are operating your infrastructure without an audit trail — and that is the gap CNCF put a name to on August 4, 2026, when StackGen principal engineer Sabith K Soopy published "You can't debug what you can't see — Observability for AI Agents," drawn from months of running agents in production.
The fix, up front in one table: every action an agent takes against your platform should land in an append-only record with these six columns filled. Here are three populated rows — a harmless read, a deploy, and the 3am rollback:
| agent / session | action (tool) | target platform object | reason (intent source) | approval | cost |
|---|---|---|---|---|---|
| release-bot / sess-8f2a | get_deploy_status (read-only) | deploy dpl-91c4, service api | plan step 2 "verify green before promoting" | auto (read-only policy) | 1.2k tokens |
| release-bot / sess-8f2a | promote_deploy | deploy dpl-91c4 → live | plan step 4 "promote after checks pass" | human approved, ticket appr-552 | 4.7k tokens |
| night-ops / sess-3d77 | rollback_service | service api, dpl-91c4 → dpl-90be | loop-recovery rule rb-3 after 3 failed health checks | auto (rollback policy, latency 800ms) | 12.3k tokens |
The rest of this post earns that table: why agents fail in ways your APM cannot see, the three observability pillars that actually work, why agent traces are noise without machine-readable platform state to join them to, and the gating checklist to run before any agent gets production access.
Agents don't fail like services
Soopy's opening line is the whole problem in one sentence: agents don't crash with stack traces. They loop, hallucinate, burn tokens, and produce plausible-looking output that is subtly wrong. A service fails by throwing; an agent fails by confidently doing the wrong thing at full speed, then doing it again.
Traditional APM, he writes, can't tell you why your agent spent far more than usual asking the same question three times. Your dashboards will show a healthy platform with normal latency while an agent burns through its budget re-invoking the same tool — because from the infrastructure's point of view, nothing is broken. The failure lives one layer up, in the agent's decision loop, where your existing telemetry has no visibility at all. That is the "can't see" in the title: not missing metrics, but a missing layer.
This is also why the failure taxonomy matters more than any single tool choice. Loops (same tool, same arguments, no progress), hallucination spirals (each wrong step becoming the next step's context), model-routing errors, and unbounded context accumulation all present identically from below — slightly elevated API usage — and completely differently from above, where the session trace shows the pattern in seconds. If you take one idea from this post, take that one: observe the decisions, not just the calls.
The three pillars, wired with OpenTelemetry
The CNCF post organizes production agent observability into three pillars — traces, costs, audit — and the timing matters: OpenTelemetry itself graduated inside CNCF in late July 2026, so the vendor-neutral plumbing this prescribes is now the stable default, not a bet. A sibling convention layer (OTel's GenAI semantic conditions, covered on this blog in July) already fixes the field names — gen_ai.tool.name, token-usage attributes, span shapes for tool calls. What follows is what to build with them.
Traces: the session timeline. Every agent session should produce an agent trace — not a generic APM trace, but the full decision history: each model call, each tool invocation, each sub-agent delegation, with timing and cost attached. Soopy's team uses Langfuse as the trace backend, with sub-agent work nested as child spans so delegation never loses its thread.
Two engineering details carry real weight. First, trace delivery must be non-blocking: tool execution should never wait on a synchronous POST to a tracing backend, so spans buffer through a batch exporter and flush periodically. Second, on shutdown, drain what remains — and if the backend is unreachable, you lose telemetry, never availability. Observability that can take down the operator is worse than none.
Costs: the unit economics. Token spend is the unit economics of agents, and it needs visibility at two levels: per session (total cost, token breakdown, which model did what) and per agent over time (daily burn rate, session counts, cost trends). The reason is geometric: an agent stuck in a loop burns tokens multiplicatively, and without cost monitoring you discover it on the invoice, not at onset.
The post's sharpest operational advice lives here — reactive alerting alone is too slow, because a tight loop in a parallel agent can burn budget in seconds before any webhook fires. The first line of defense is pre-flight circuit breakers: hard iteration caps, per-tool call budgets, and loop detection that blocks identical consecutive calls.
Alerts are the second line: fire when a single session exceeds a multiple of that agent's rolling-average cost, which catches the slow burners — hallucination spirals, routing errors, creeping context — that slip past hard limits.
Audit: the immutable record. Every tool call, governance decision, and memory operation goes into an append-only record — structured, timestamped, searchable — with sensitive tool outputs sanitized before logging. That last clause is doing heavy lifting: the audit trail gets queried during incident reviews, and if it contains credentials or PII, your observability system becomes a liability. Redaction at write time is non-negotiable.
And the split that keeps the whole thing affordable: traces are for debugging, metrics are for alerting. Export bounded metrics to Prometheus — tool success and failure rates by tool name, per-agent session costs, approval-latency histograms — and keep labels low-cardinality. Tool names and agent names are safe; they have bounded values. Never put session IDs in Prometheus labels. A fleet running thousands of agent sessions a day will cardinality-crash its metrics server, and per-session detail belongs in the trace backend anyway.
Why traces are noise without machine-readable platform state
Here is the step the CNCF post implies and a PaaS operator has to make explicit. An agent trace that says "called rollback_service at 03:12, cost 12.3k tokens" is only half an audit record. The other half is the platform side: which service, which deploy revisions, which API request ID, what the rollout state was before and after.
Without that join, you have agent telemetry floating next to platform telemetry with no key between them — noise with timestamps.
This is why the "machine-readable platform state" clause in the premise matters. A platform with a Render-compatible API surface gives every agent action a natural join key for free: the deploy ID, the service ID, the request ID the API already returns. The six-column table at the top of this post works because columns two and three (action, target) come from the agent's span attributes while columns three and four also resolve to platform objects with their own independent history. When the 3am rollback row says dpl-91c4 → dpl-90be, you can click from the agent's "why" straight into the platform's record of what those two deploys contained and who shipped them. Neither side alone answers "which agent changed what, when, and why" — the answer lives in the join.
The vocabulary post and this post are two halves of one system, to put it plainly. Standard span fields give every agent's traces the same shape; a machine-readable platform gives those traces something to point at. Build the first without the second and you can parse every span perfectly while still unable to reconstruct a single incident.
The gating checklist before prod access
None of this is useful as philosophy. As a gate — no agent touches production until every line checks out — it looks like this:
- Append-only audit log. Every tool call, governance decision, and memory write lands structured, timestamped, and searchable, with credential and PII redaction at write time. Query it for last week's deploys before you trust it with tonight's.
- Approvals with latency metrics. Mutating actions need an approval path, and approval latency gets its own histogram — stale approvals mean blocked agents, and blocked agents get worked around. The rollback row in the opening table carried an 800ms-latency auto-approval under a named policy, not a missing approval.
- Automated session review. With hundreds of sessions daily, humans review flags, not sessions. Score every completed trace on duration, cost, tool count, loop signals, and token efficiency; flag loops, cost outliers, and tool-error clusters for human review.
- A doctor command. One command that checks model connectivity, vector-store reachability, pending approvals, memory counts, trace-backend status, and integration health — clear pass or fail per dependency. Soopy's team reports this saves more time than any dashboard, because most "agent is broken" incidents are really "agent dependency is down."
- The cost canary. Alert on session cost versus the agent's rolling average first; debug second. Sudden spend is almost always a bug, and it is the fastest signal you have.
Then watch the short list that tells you the system is working: session cost against rolling average (catches loops early), identical consecutive tool calls (loop detection before cost explodes), approval latency (blocked agents), model error rate (provider issues versus agent bugs), daily token burn against budget (invoice surprises), and audit-log growth rate (runaway execution shows up here first).
What "production-ready agent" actually means
The honest redefinition buried in all of this: an agent is production-ready not when its success rate looks good in evals, but when its failures are reconstructible. Loops, hallucinations, and token burn are going to happen — the question is whether each one leaves a trace you can follow, a cost record you can bound, and an audit row you can query. Cost is the canary, traces are the debugger, metrics are the alerter, and the audit log is the record of truth the other three point into.
For a platform built on the premise that AI agents operate infrastructure like humans do, this isn't a nice-to-have layer — it is the precondition. A human operator without shell history, deploy logs, and an approval trail would never be handed production access; an agent without the same three should get no better deal. Build the trail first, grant the access second, and the 3am incident becomes a five-minute query instead of a five-hour mystery.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with AI agents as first-class operators. Star the repo on GitHub or deploy your first app today.
