Dora NodaTop coding agents now resolve roughly three out of four real GitHub issues on SWE-bench Verified — and then score barely one in two on Terminal-Bench, the benchmark that asks them to do ordinary terminal work: compile code, configure a server, recover a broken environment. Same models, same year, a gap of twenty-plus points that has survived every model generation since both boards existed.
Here is the verdict up front, because the rest of this post is just the receipts: if you are deciding how an AI agent should operate your infrastructure, do not give it a raw shell with kubectl and SSH and hope for the best. The terminal session is exactly the task class agents are worst at — long, stateful, full of silent failures. Give it scoped, structured tools instead: deploy, rollback, scale, each one a typed action with a preview and an audit trail. A ten-step shell session collapses into one well-defined call, which happens to be the exact shape of task agents already excel at.
That gap is not a temporary embarrassment. It is a design constraint, and platforms that build around it will have agents they can trust on call. Platforms that ignore it will have a very fast junior operator with root access and no supervision.
What each benchmark actually measures
The two boards look similar from a distance — both grade agents on real software work — but they test almost opposite skills.
SWE-bench Verified is 500 human-validated bug-fix tasks drawn from popular Python projects: Django, SymPy, scikit-learn, pytest, Flask. The agent gets an issue description and a repository, writes a patch, and the grade is binary: do the project's own tests pass. It is single-shot, stateless work with a crisp oracle. As of mid-2026, frontier coding agents clustered at 74–79% on the board — and by late 2026 headline numbers had climbed into the 90s, which says as much about benchmark saturation as about capability.
Terminal-Bench 2.0 is 89 hard tasks executed inside isolated Docker containers through the open-source Harbor evaluation framework, built by a Stanford and Laude Institute team with more than 85 contributors and released in late 2025. Each task is a slice of real professional work: compile an unfamiliar codebase, train a model, set up a server, debug a legacy system, convert COBOL to Python, administer packages and files. The agent reads a natural-language instruction, plans across dozens of shell commands, recovers from its own errors, and produces a verifiable end state graded by an outcome script. Early-2026 direct model scores clustered at 52–58% for the same agent class scoring in the mid-70s on SWE-bench; newer agent-plus-harness combinations have since pushed past 80%.
A note on the numbers, because honesty matters here: both boards move. Absolute scores climb as models and harnesses improve, and any post that freezes one month's leaderboard as eternal truth is lying by omission. What has not moved, across every generation both benchmarks have coexisted, is the ordering: the same system scores materially lower on autonomous terminal work than on code editing. The gap is the finding. The exact point spread is weather.
Why is the terminal so much harder? A patch is one artifact with one oracle. A terminal task is a twenty-step plan executed against mutable state, where step 14 can silently invalidate steps 3 through 9, the error message lies about which step failed, and nobody tells you when you are done. SWE-bench measures whether agents can act correctly once. Terminal-Bench measures whether they can sustain correct action across a horizon — plan, execute, notice drift, recover, verify, stop.
How agents actually fail at the terminal
The failure analyses are unusually consistent, and they read like an on-call incident review. Trajectory-level studies of Terminal-Bench runs keep finding the same four modes: agents disobey the specification halfway through, repeat steps they already completed, miss termination conditions and wander past the goal, and — the finding of the 2026 long-horizon follow-up work — string together many locally correct actions without ever converting progress into a finished artifact before the budget expires. The bottleneck, in that paper's phrasing, is not local execution correctness but long-horizon completion: sustaining progress, verifying it, and stopping.
Map each of those onto deploy operations and the blood drains from your face a little, because every one of them is a production incident wearing a benchmark costume:
- Disobeying the spec mid-run is the agent that was told to roll back the canary and instead re-applies the broken manifest because the file was still in its context window.
- Repeating completed steps is the agent that restarts a migration it already ran, because it never checked whether the first attempt committed.
- Missing termination is the agent that keeps
kubectl rollout restart-ing a Deployment whose new pods are CrashLooping, waiting for a healthy state that its own restarts keep postponing. - Never verifying completion is the agent that reports "deploy succeeded" when the pipeline exited zero but the pods never became ready — the single most expensive sentence in agent operations.
None of these is a knowledge problem. The agent knows what a rollout is. They are all horizon-management problems: keeping the goal, the state, and the stop condition aligned across many steps in a mutable world. That is precisely what raw shell access maximizes — maximum steps, maximum mutability, minimum structure — and precisely what a scoped tool call minimizes.
The harness matters more than the model
There is a second literature that points the same direction, and it is arguably the more important one. A 2026 study of the "scaffold effect" in coding agents measured a 40x performance gap across harnesses on terminal tasks with the model held constant — the scaffolding that issues tools, manages context, and decides when to stop mattered more than which frontier model sat inside it. Agentic Harness Engineering work lifted Terminal-Bench pass rates from 69.7% to 77.0% by evolving the harness alone, with the gains transferring to other models untouched. Teams letting agents rewrite their own harness rules reported improvements up to 60%.
Read that twice, because it inverts the default platform instinct. When agent operations fail, the instinct is to wait for a smarter model. The evidence says the lever is the harness: constrain the action space, manage the context, own the stop condition. A structured tool surface is not a limitation you bolt onto a capable agent. It is the capability — the harness is the product.
The most elegant confirmation comes from the other direction. A 2026 result that made the rounds showed a two-cent-per-call model scoring 78.2% on SWE-bench Verified — ahead of models costing nearly forty times more — with the difference attributed to architectural context injected through MCP: the cheap model won because its harness handed it structured understanding of the codebase instead of making it reconstruct that understanding from raw files. Structure beat scale. It usually does.
Before and after: rolling back a bad deploy
Enough theory. Trace one concrete, utterly typical task — "the canary is erroring, roll it back" — through both architectures and count the failure points.
The raw-shell path is roughly ten steps, each one a place for the four failure modes above to bite. SSH into the right bastion or select the right kubeconfig context. Identify which release is live and which is the canary. Confirm the error rate from logs or metrics. Find the previous stable revision. Render or retrieve its manifest. Diff it against live state so the rollback does not also revert an unrelated config change someone applied by hand. Apply it. Watch the rollout. Verify pod readiness and error rates, not just pipeline exit code. Decide you are done and stop. Every step emits unstructured text the agent must parse correctly; three of them mutate production; two of them have silent failure modes where success looks identical to failure until a customer tells you otherwise.
The structured path is one call:
rollback(service="checkout", target="last-stable", reason="canary 500-rate > 2%")with a typed response the agent can actually reason about: previous revision pinned by the platform, hand-applied drift surfaced rather than silently kept or clobbered, rollout watched by the tool rather than by the agent polling kubectl get pods in a loop it may never exit, readiness verified against the platform's own health gates before the call returns success. The ten failure points did not get smarter agents assigned to them. They stopped existing — absorbed into deterministic code, which is where deterministic work belongs.
This is the sense in which a scoped tool "collapses a multi-step terminal session into the single well-defined action class agents already excel at." SWE-bench tasks and MCP tool calls share a shape: bounded inputs, a clear goal, a machine-checkable outcome. Raw terminal sessions share nothing with that shape. The benchmark gap is, among other things, a measurement of that shape mismatch.
A minimal deploy-agent tool surface
If the argument above holds, the design falls out directly: enumerate the operations an agent may perform, type them tightly, and refuse everything else. Five tools cover the overwhelming majority of deploy-from-chat traffic:
deploy(service, revision, environment)— builds or promotes exactly one revision to exactly one environment. Guardrail: immutable revision pin, never "latest"; returns the rollout handle, not a vibes-based summary.rollback(service, target)— the one-call path from the previous section. Guardrail: target must resolve to a previously healthy revision; dry-run preview shows what changes before anything mutates.scale(service, replicas)— bounded integers, per-service ceilings set by a human. Guardrail: refuses orders of magnitude jumps without confirmation; this is the tool that turns a typo into an incident if untyped.logs(service, since, filter)— read-only, paginated, redacted. Guardrail: secrets scrubbed at the tool layer, because the agent's context window is the worst secrets manager ever built.status(service)— the stop condition, as a service: revision, readiness, error rate, rollout state in one typed response. Guardrail: none needed — this is the cheapest reliability win on the list, since "verify completion" is the failure mode agents hit most.
Three properties cut across all five. First, typed inputs and outputs — the agent fills parameters, not parses prose, which removes the largest single error class in terminal trajectories. Second, preview before mutation — every write path shows its plan, because agents that cannot reliably verify completion must not be the only thing standing between a sentence and production. Third, an audit log owned by the platform — who asked, what was called, what changed — because "the agent did it" is not an incident timeline.
Note what is absent: no shell tool, no kubectl passthrough, no SSH. That is deliberate, and it is the entire thesis. The ecosystem already ships Kubernetes MCP servers that wrap kubectl, Helm, and ArgoCD behind natural-language-friendly tools, and control planes for hosting them — useful, but wrapping a forty-verb CLI one-to-one preserves the action-space sprawl the benchmarks punish. Fewer, higher-level, opinionated tools beat a complete CLI mirror. The DevOps use case is real — Kubernetes troubleshooting eats roughly a third of on-call time in surveys — which is exactly why the interface to it deserves the design effort, not a passthrough.
The platform conclusion
Step back and the story is simple. Agents crossed the "writes code like a strong junior" threshold and the benchmarks prove it. They have not crossed the "operates infrastructure unsupervised" threshold and the benchmarks prove that too — the gap between the two boards has survived every model release so far. Waiting for scale to close it misunderstands the failure: the failures are horizon management, and horizon management is a property of the harness, not the weights.
So build the harness. Constrain the action space to typed tools. Own the stop condition with a status call. Keep deterministic work in deterministic code and let the agent do what it is already good at: mapping a human sentence onto one well-defined action. When the next model generation arrives, your agents get smarter inside a surface that was already safe. That is the bet — not that agents will stay clumsy at terminals forever, but that a platform should never have needed them to be graceful there in the first place.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Machine-readable operations are a first-class surface, not an afterthought: star the repo on GitHub or deploy your first app today.
