Your APM Says the Deploy Was Green. The Agent Retried the Migration Three Times

Your deploy dashboard says last night's rollout took four minutes and succeeded. What it doesn't say: the AI agent running it invoked fourteen tools, retried the database migration three times, burned about $2.40 in tokens, hit a rollback decision point once — and chose to continue. Every one of those facts mattered more than the green checkmark. None of them is in your APM.

That is the argument the CNCF's August 2026 post on observability for AI agents makes concrete: request-scoped APM is built on three assumptions — deterministic call graphs, bounded traces, meaningful p99 latency — and an agent system that plans, branches, retries, and calls tools in loops violates all three at once. An agent can return HTTP 200 with low latency and zero errors while looping, misusing tools, or confidently producing nonsense. Your existing dashboards will call that a healthy request.

So what does the healthy alternative actually look like? Here is the trace your platform should have emitted for that four-minute deploy — one root span per run, every decision a child:

deploy run #4821  ·  4m02s  ·  214,300 tokens  ·  \$2.41  ·  outcome: success*
├─ invoke_agent deploy-agent ............ 4m02s  gen_ai.request.model="claude-opus-4-6"
│  ├─ execute_tool plan_rollout ......... 12s    args={target: prod, strategy: rolling}
│  ├─ chat turn 1 (assess) .............. 18s    gen_ai.usage.input_tokens=18,204
│  ├─ execute_tool migrate_db ........... 41s    status=ERROR attempt=1 err="lock timeout"
│  ├─ execute_tool migrate_db ........... 38s    status=ERROR attempt=2 err="lock timeout"
│  │   └─ event: retry_budget 2/3 consumed · backoff=30s
│  ├─ execute_tool migrate_db ........... 22s    status=OK   attempt=3
│  ├─ chat turn 4 (rollback decision) .... 9s    decision=continue reason="migration verified"
│  └─ execute_tool rollout_restart ...... 71s    status=OK   strategy=rolling
└─ *success with 2 failed attempts, 1 near-rollback, and a token bill attached

Three things to notice. First, the interesting unit is the run, not the request: a four-minute session containing dozens of model calls and tool invocations. Second, cost is a first-class span attribute — tokens in, tokens out, dollars per run — because for agents, spend is a reliability signal: a run burning 10x its median tokens is usually a run stuck in a loop. Third, the rollback decision is recorded as an event with its reasoning, because "the agent considered rolling back and didn't" is exactly the fact you will need at the next incident review. A traditional trace shows you none of this: it shows one successful HTTP request to the deploy endpoint.

This post works through why that gap exists, what the converging standard vocabulary for closing it looks like, which tools already speak it — including ones you can self-host — and what a platform has to emit natively so the trace above is not aspirational.

Why all three APM assumptions break at once​

Traditional APM answers three questions: how slow (latency percentiles), how broken (error rates), how much (throughput). Each rests on an assumption that holds for request-response services and fails for agents.

Assumption 1: the call graph is deterministic. In a microservice trace, checkout → charge → fulfill happens in that order every time, so a span waterfall that deviates from the shape is itself the anomaly. An agent's "call graph" is decided at runtime by the model: the same deploy prompt can produce fourteen tool calls tonight and four tomorrow, in a different order, with different arguments. There is no canonical shape to diff against.

Deviation-from-shape alerting — the backbone of trace-based APM — has nothing to hold onto. What you need instead is trajectory observability: the sequence of reasoning step → tool call → observation, recorded as it happened, evaluable after the fact. That is why the ecosystem is converging on run-scoped traces (one root invoke_agent span per run) rather than request-scoped ones.

Assumption 2: traces are bounded. A web request fans out to a knowable set of downstream calls and finishes. An agent loop has no static bound: a retry storm, a planning loop that never converges, or two subagents delegating back and forth can extend a run indefinitely while every individual span looks fine. Practitioners have learned to watch proxies for unboundedness instead — context-window fill (average and p99), frequency of context-trimming events, tool-call counts per run, tokens per successful task rather than tokens per call. Rising context usage foreshadows failures the way rising queue depth foreshadows an outage: it tells you the loop is going nowhere before it gets there. None of these are APM metrics. All of them are cheap to emit.

Assumption 3: p99 latency means something. For a request, slower is worse, monotonically. For an agent run, a four-minute run can be a careful, correct rollout and a forty-second run can be a confidently wrong one that skipped verification. Latency and quality are decorrelated — the failure mode the field keeps rediscovering is the agent that returns HTTP 200 in 400ms with a fluent, wrong answer.

So the "error rate" you actually need is a quality signal: did the migration verify, did the rollout converge, did a judge (human or model) accept the outcome. Production agent teams pair traces with evaluation scores on the same run ID — Arize's handbook frames it as capturing framework traces, evaluating spans, traces, trajectories, and sessions, and cutting datasets from production failures. The SLI for an agent operator is task success per run, with cost per successful task as the efficiency denominator. Token spend alone never tells you whether the agent is working.

The through-line: APM observes the outside of a system (requests in, responses out, latency between). Agents have to be observed from the inside (decisions, tool calls, spend, retries), because the outside is a single green checkmark either way.

The vocabulary is converging: OTel GenAI semconv + MELT​

The good news is you do not have to invent this vocabulary. Three 2026 developments settled it.

First, OpenTelemetry graduated as a CNCF project in May 2026 — the second-highest-velocity project in the ecosystem after Kubernetes itself — cementing OTel as the vendor-neutral telemetry substrate. If your agent emits OTel, every conformant backend ingests it with no SDK changes.

Second, the GenAI semantic conventions give agent spans standard names and attributes. The shape mid-2026 (still experimental, behind OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, so pin the version):

The waterfall in this post's opening is just these conventions rendered as ASCII. Emit them and you get every conformant backend for free — the major APM vendors now ingest the GenAI conventions natively, alongside the open-source backends below.

Third, the CNCF's cloud-native agentic standards (March 2026) name the observability requirement explicitly: a standard MELT stack — metrics, events, logs, traces — consolidated so the system stays explainable and debuggable, plus network flow logs for the security side. "Events" earning equal billing with the other three is the tell: for agents, discrete structured occurrences (tool called, retry exhausted, rollback considered) carry as much signal as any time series. If your platform's deploy pipeline only emits logs and exit codes today, the standards checklist is telling you what to add.

What to run: three backends that already speak agent​

For a team that self-hosts, the tool choice reduces to one question: where do the traces land, and who operates the landing zone?

The practical move for a small platform team: keep Jaeger if it is already deployed, stand up Langfuse or Phoenix next to it for the agent-quality layer (evals, cost, trajectories), and join them on the run ID. Do not build a bespoke trace UI — Headlamp's CAPI-plugin story already taught this lesson for cluster state, and it applies doubly to traces, where every backend above renders the invoke_agent waterfall out of the box.

One integration detail worth knowing if you run MCP servers: Arize's MCP tracing setup requires verbose=False on the Python server so the banner doesn't corrupt the stdio wire protocol — the kind of sharp edge that confirms tool-call spans must be emitted server-side, by the platform, not scraped from client logs.

The platform lesson: emit machine-readable state natively​

The thesis that motivated this post survives contact with the research intact: you cannot bolt observability onto an operator whose actions you never recorded. An external APM agent watching HTTP status codes cannot reconstruct which tools the deploy agent called, in what order, for how many tokens, and which rollback it considered. Only the platform that executed — or brokered — those calls can emit that record. If your PaaS wants agents as first-class operators, the event stream is not a nice-to-have; it is the API the operator reads back.

Concretely, a git-push platform needs five emissions before its first agent-run deploy, not after:

1. Server-side spans for every brokered tool call. The platform invokes (or proxies) the deploy, the migration, the restart — so it emits the execute_tool span with arguments, attempt number, and result. Client-side logging is lossy by construction.
2. One root run ID across model, tools, and infra. The invoke_agent span links the LLM turns, the MCP tool calls, and the Kubernetes events (pod restarts, Job completions) into a single trace. Without the shared ID you have three dashboards and no story.
3. A per-run token and cost ledger. Record gen_ai.usage.* per turn and roll it up per run. Alert on cost per successful task, not raw spend — a 10x-token run that succeeded is interesting, a 10x-token run that failed is the incident.
4. Decision events with reasons. Retries, backoff, rollback-considered-and-declined, human-approval gates: each a structured event carrying the why. This is the record the post-incident review reads.
5. Trajectory evals on production runs. Sample real runs into eval datasets, score them, and feed failures back as regression cases. The SLI is task success per run; the eval suite is how you know it moved.

Note what is not on the list: a new APM vendor, a new dashboard, a new percentile. The list is five emission points in systems you already operate. The MCP stateless-core shift (July 2026 spec) makes this easier, not harder — stateless tool servers push session/run correlation up to the broker, which is exactly where your platform sits.

The green checkmark is not the deploy​

The CNCF post's real claim is narrower and more useful than "APM is dead": request-scoped monitoring answers did the endpoint respond, and agent operations need what did the operator do. Those are different questions, they need different spans, and the second set has to be emitted by the platform that runs the tools — standard vocabulary provided by OTel's GenAI conventions, storage provided by backends you can already self-host, quality signal provided by evals on real trajectories.

So the next time your dashboard shows a four-minute green deploy, ask for the run trace: fourteen tool calls, three migration attempts, one declined rollback, $2.41. If your platform can't produce it, you didn't observe a deployment. You observed a checkmark.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.