Dora NodaA deploy/rollback MCP tool that gets confused rolls one tenant's app back to last week's build. Annoying, but a git push fixes it. An upgrade_cluster MCP tool that gets confused is a different animal: it can leave a fleet with some nodes running Kubernetes 1.35 and others still on 1.32, mid-drain, with a PodDisruptionBudget blocking the next eviction — and nobody at a terminal to notice. The deploy/rollback tool contracts this list has already covered (rollback-safety guardrails, scoped credentials, dry-run diffs) are all scoped to a single tenant's app. A fleet-wide Cluster API node upgrade is the next, higher-blast-radius operation an agent could plausibly be handed, and it needs a narrower tool contract than "give the agent kubectl."
How Cluster API Actually Rolls a Node Upgrade Today
Before designing a tool contract, it's worth being precise about what the underlying operation actually does, because the safety design has to sit on top of real mechanics, not a simplified mental model.
A Cluster API-managed cluster has two upgrade surfaces, and they behave differently:
- Control plane — a
KubeadmControlPlaneresource. Bumpingspec.versiontriggers a rolling replacement of control-plane Machines, one at a time by default, because a control plane can't safely surge past its etcd quorum size the way a stateless worker pool can. - Workers — one or more
MachineDeploymentresources. Bumpingspec.template.spec.versiontriggers a rolling replacement governed bymaxSurge/maxUnavailable, the same knobs a KubernetesDeploymentuses for pods, just one layer down at the machine level. The Cluster API Book's own upgrade guide is explicit that control plane and workers are upgraded as two separate steps, control plane first — a worker can run up to three minor versions behind the API server under Kubernetes' version skew policy, but the API server can never run behind its own workers.
Each replacement Machine goes through the same sequence: a new machine is provisioned, joins as a NotReady node, and only becomes eligible to receive workloads once kubelet reports ready. The old machine is cordoned, drained — evicting pods while respecting any PodDisruptionBudget that covers them — and deleted only after the drain succeeds. Cluster API's MachineHealthCheck watches the new node against a nodeStartupTimeout; if kubelet never reports in, the Machine gets remediated (replaced) automatically. None of this is exotic — it's the same rolling-update primitive Kubernetes uses everywhere — but it has two structural failure points worth naming up front, because they're exactly what an agent-facing tool has to guard against: a drain that stalls forever because a PodDisruptionBudget never allows the last eviction, and a fleet that ends up straddling the version-skew boundary if the upgrade stops partway through.
Designing upgrade_cluster and rollback_cluster
A human running this upgrade by hand does it in two deliberately separated steps — bump the control plane, confirm it's healthy, then bump the workers — and watches kubectl get machines scroll by between each replacement, ready to kubectl rollout pause the moment something looks wrong. An MCP tool pair that hands the same authority to an agent has to encode that same discipline as hard contract, not as something the model is expected to infer from a prompt. Here's what that contract needs, concretely.
Scope the tool to one MachineDeployment or control plane at a time — never "the cluster." upgrade_cluster should take a single target reference (a specific KubeadmControlPlane name, or a specific MachineDeployment name) and a target version, not a cluster-wide "bump everything" call. This isn't a convenience choice — it's what makes the control-plane-before-workers ordering enforceable at all: if control plane and worker upgrades are two separate tool calls against two separate objects, the server can reject a worker-upgrade call outright when the control plane isn't yet on a compatible version, instead of trying to sequence an implicit multi-object operation correctly inside one call.
A plan mode before any execute call. The app-level rollback pattern this list already covers leans on a dry-run diff — a preview the agent (and the human watching the agent) can read before anything changes. A multi-node rolling upgrade needs the equivalent, but it's a richer artifact than a single-deploy diff: upgrade_cluster(mode: "plan", ...) should return the ordered list of Machines that will be replaced, the surge/unavailable count that governs how many run concurrently, and the version-skew window the fleet will sit in mid-rollout (e.g., "workers will run 1.32 and 1.35 simultaneously for the duration of this rollout, within the 3-minor-version skew limit"). Only mode: "execute" against an unchanged plan actually starts replacing machines.
Cap maxSurge/maxUnavailable server-side, not agent-side. The tool schema shouldn't let the model set these to arbitrary values — the server enforces a fixed ceiling (say, maxSurge: 1, maxUnavailable: 0 as the default, matching the "don't drop capacity mid-upgrade" posture most production fleets want) and only allows the agent to select from a small enumerated set of pre-approved profiles, if that. An agent that's confused about urgency shouldn't be able to set maxUnavailable: 100% and take the whole pool down at once.
A per-node health gate between every replacement, not just at the end. The orchestration layer behind upgrade_cluster shouldn't fire off the whole MachineDeployment bump and walk away — it should advance one Machine at a time, waiting for the new node to clear MachineHealthCheck and hold Ready for a soak window (mirroring minReadySeconds) before triggering the next replacement. This is stricter than what raw Cluster API reconciliation does on its own; it's the MCP server's job to gate its own advancement decisions on health, the same way a careful human pauses between machines to eyeball kubectl get nodes instead of trusting the rolling update to run unattended end to end.
A hard stop that isn't a retry loop. If a replacement node fails to rejoin within nodeStartupTimeout, Cluster API's own MachineHealthCheck will remediate it — but remediation means "try again," and an agent watching that happen has no natural instinct to stop trying after N failures the way a human on-call engineer does. The tool server needs its own ceiling, independent of CAPI's: after a fixed number of remediation cycles on the same Machine, or after the fleet has spent longer than a fixed budget straddling two kubelet minor versions, upgrade_cluster auto-pauses the MachineDeployment (the same spec.rollout pause CAPI already supports) and returns a status that requires explicit human acknowledgment before any further tool call can resume it. That ceiling is the actual answer to "a hard stop on partial-fleet state an agent can't reason its way out of" — a fleet frozen mid-upgrade is a state a human can safely inspect; an agent stuck in an unbounded remediate-and-retry loop is not.
rollback_cluster inherits the app-level pattern, with one worker/control-plane asymmetry called out explicitly. For workers, the same guardrail the sister post on app-level rollback already established applies here: the tool accepts target: "previous" or an explicit version string that's already present in the MachineDeployment's own revision history — never an agent-computed offset. For the control plane, the tool has to refuse a different class of request entirely: Cluster API does not support downgrading a KubeadmControlPlane's Kubernetes version — the etcd schema and API server data model only move forward. "Rollback" for a control plane means restoring from a pre-upgrade etcd snapshot, a break-glass operation with its own blast radius, not a parameter rollback_cluster should expose as a same-shape sibling of the worker case. The tool contract should say so out loud — a rollback_cluster call against a control-plane target returns an explicit unsupported error naming the snapshot-restore procedure, rather than silently attempting something CAPI can't do.
An audit log that records which actor drove which Machine replacement. Every Machine created or deleted as part of an agent-initiated upgrade should carry an annotation identifying the tool call and the calling agent's identity, distinct from a human-initiated kubectl or GitOps-triggered rollout. When a fleet-wide version bump goes wrong at 2 a.m., "was this us or the on-call agent" needs to be answerable from kubectl describe machine, not reconstructed from a chat log.
Why "The Same Objects a kubectl Session Already Manages" Isn't Enough
It's tempting to conclude that none of this is necessary — Cluster API already exposes MachineDeployment, KubeadmControlPlane, and MachineHealthCheck as ordinary Kubernetes objects, and a human with kubectl access can already do everything described above by hand. If a human can safely run the upgrade with nothing but kubectl edit and patience, why does an agent need a narrower tool?
Because the human's safety doesn't come from the objects being safe to edit — it comes from everything the human does around the edit that the raw API doesn't enforce: watching kubectl get machines -w between each replacement, noticing a stuck drain before it compounds, and stopping — not retrying harder — when something looks wrong. That's the same confused-deputy shape this list has already named for app-level rollback: a system holding real privileged credentials, executing a request without independently verifying whether this specific request is one it should honor at this privilege level. For a single app's deploy, the cost of getting that judgment wrong is a bad release, fixed by another push. For a fleet-wide version bump, the cost of getting it wrong is a cluster stuck straddling a version-skew boundary with some fraction of its nodes drained and no clean, symmetric way back — control-plane state that, unlike a container image, doesn't roll backward on request. The kubectl objects being the same is exactly why the tool contract has to be narrower: the objects were never what made the human-operated version safe.
Bex.co is the open-source, AI-native Render alternative, built on Cluster API underneath a git-push, Render-compatible surface — a platform an AI agent can operate through tool contracts scoped no wider than the operation actually needs. Star the repo on GitHub or deploy your first app today.
