Skip to main content

Your Deploy CLI Should Behave Like kubectl: Kubeconfig Merges, Impersonation, and Agent-Operated Fleets

11 min readDora NodaDora Noda
Share
On this page

Every platform team eventually writes its own CLI. And almost every one of them reimplements — badly — a contract that kubectl settled a decade ago: how a tool figures out which cluster it is talking to, who it is, and whose shoes it is standing in. The symptoms are always the same. A deploy script that works on your laptop and authenticates against the wrong fleet in CI. A support runbook that tells operators to juggle three kubeconfig files with cp commands. An AI agent, handed fleet access through a shiny new MCP server, that debugs tenant staging and restarts tenant production — because nothing in the tool chain told it the difference.

Here is the contract, stated up front so the rest of this post is just substantiation. When a kubectl-shaped tool starts, it resolves its identity through exactly three steps: if a --kubeconfig flag names a file, only that file is loaded and no merging happens; otherwise, if the KUBECONFIG environment variable is set, its file list is merged in order; otherwise, ~/.kube/config is used alone. Merge precedence inside that list is first-wins for named entries, the first file that sets current-context picks the starting context, and explicit flags beat everything the files say. That paragraph is the whole article in miniature. The rest shows you the merge table with a worked example, the three footguns that bite the moment an agent starts writing kubeconfig, the impersonation primitive that replaces per-tenant credential juggling, and the ten lines of Go that buy your CLI all of it for free.

The resolution chain: three steps, no more

The loading rules live in client-go's clientcmd package — NewDefaultClientConfigLoadingRules builds the chain, and it is the same chain kubectl documents on its organizing-access reference page. Step one: the --kubeconfig (equivalently, ConfigFlags.KubeConfig) value, when set, is the entire universe. One file, no merging, which is exactly what you want for automation that must not be influenced by whatever else happens to be on the machine. Step two: the KUBECONFIG environment variable, a list of paths separated by colons on Linux and macOS and semicolons on Windows, merged front to back. Step three, the fallback when neither is set: ${HOME}/.kube/config, alone, unmerged.

Two details in that chain do most of the operational work. First, a missing file in the KUBECONFIG list is skipped, not fatal — the chain is explicitly allowed to name files that do not exist yet — while a file that exists but cannot be deserialized is a hard error. Garbage fails loudly; absence fails silently. Second, writes go somewhere specific: when a value is created by kubectl config, it lands in the first file in the chain that exists, or the last file in the chain if none exists yet. kubectl config use-context is not editing "your kubeconfig" in the abstract; it is editing the first file the loader would have read. Hold that thought — it becomes footgun number two below.

The merge table: what wins when two files disagree

Merging is not concatenation; it is an ordered overlay with per-key precedence. The rules, per the upstream docs and the kubectl config reference:

DecisionWinner
Which context to start from--context flag if given, else current-context from the first file in the chain that sets one
Which cluster a context points at--cluster flag if given, else the cluster entry of the resolved context, looked up in the merged map where the first file defining that name wins
Which user to authenticate as--user flag if given, else the user entry of the resolved context, same first-wins map lookup
Which namespace--namespace flag if given, else the context's namespace, else default
Duplicate cluster/user/context names across filesFirst file in the chain that defines the name wins; later definitions are shadowed but harmless
Server, certs, tokens for a given nameTravel with the winning entry — you cannot take the cluster address from file A and the credentials from file B under the same name

Make it concrete. Suppose file A, the platform-issued admin config, sets current-context: fleet-admin and defines a cluster named mgmt pointing at the management API with admin credentials. File B, generated by a tenant's onboarding job, defines a context tenant-a whose cluster entry is also named mgmt but points at a regional endpoint with tenant-scoped credentials. With KUBECONFIG=fileA:fileB, the resolved session starts at fleet-admin, and the name mgmt everywhere resolves to file A's definition — file B's endpoint is shadowed, silently.

Swap the order to fileB:fileA and the same bytes on disk produce a different fleet, a different identity, and a different blast radius. Nothing errors. Nothing warns. The order of a colon-separated string is a security boundary, and it is the kind of boundary humans misread and agents cannot see at all unless your tooling surfaces the resolved identity — kubectl config current-context plus kubectl config view --minify — before every mutating operation.

Three footguns, before you let an agent write kubeconfig

The merge table is read-only knowledge. The moment a tool — human-driven or agent-driven — starts writing, three behaviors surprise everyone exactly once.

Footgun one: absence is allowed, garbage is fatal. Because missing files are skipped, a KUBECONFIG chain may legitimately reference files that do not exist yet — a tenant config that has not been issued, an optional overlay. An agent that "repairs" the environment by creating every referenced path, or that treats a skipped file as an error to route around, is fixing something that is not broken. Conversely, a file with unparseable content fails the entire load. Validation for anything that generates kubeconfig — your onboarding job, your agent's bootstrap step — should therefore be syntactic (kubectl config view --kubeconfig=newfile round-trips) before the file ever joins a chain, because one bad file poisons every context in the merge, not just its own.

Footgun two: writes land in the first file that exists. An operator — or an agent following a support runbook — that runs kubectl config use-context tenant-a while KUBECONFIG=fileA:fileB is mutating file A, the platform-issued admin config, to point its current-context at the tenant. The next load with the same chain starts in a different context because an ostensibly read-scoped "switch" rewrote shared state. For interactive humans this is a papercut; for agents executing multi-step plans it is a persistent-state mutation hiding inside a step labeled "inspect". If your agent tooling needs context switching, do it per-invocation with an explicit --context flag or an isolated KUBECONFIG value, never by shelling out to config use-context against a shared chain.

Footgun three: BuildConfigFromFlags takes one file, not a chain. This is the classic multi-cluster bug, and it has shipped in real projects: clientcmd.BuildConfigFromFlags("", kubeconfigPath) treats its second argument as a single file path. Hand it a colon-separated KUBECONFIG value and the loader stats a filename containing a colon, which does not exist. The fix, demonstrated in the open by fixes like Testkube's PR #8102, is to split the value with filepath.SplitList and load it through the default loading rules — the code path that implements kubectl's own precedence — instead of the single-file shortcut. Any fleet CLI that shells single-file loading around a multi-file world will work on every developer laptop (one file) and break in exactly the production setup (many files) where it matters. Audit your codebase for BuildConfigFromFlags calls that receive environment-derived paths; each one is this bug waiting for a second cluster.

Impersonation is your act-as-tenant primitive

Support work on a multi-tenant fleet keeps presenting the same choice: mint a credential per tenant and manage its distribution, rotation, and revocation — or hold one powerful credential and narrow it per request. Kubernetes built the second option into the API server: impersonation. A subject with the impersonate verb on users, groups, or serviceaccounts can send Impersonate-User and Impersonate-Group headers, and the API server then authorizes the request as the impersonated identity. In kubectl spelling:

text
kubectl get pods -n tenant-a --as=tenant-a-support --as-group=tenant-viewers
kubectl auth can-i delete deployments -n tenant-a --as=tenant-a-support

The second line is the half operators underuse. auth can-i --as is a dry-run of someone else's RBAC: it answers "what can this tenant actually do here" without borrowing their credentials, which makes it both a support diagnostic and a preflight check an agent can run before attempting a mutating call. The grant itself is a small, auditable ClusterRole, and the audit log records both the impersonator and the impersonated identity — which is precisely the attribution you need when the "operator" is a support agent, human or software. Two limits to respect: impersonation is itself an RBAC grant, so it must be issued deliberately and narrowly (the groups that can impersonate system:masters should fit on one line, ideally an empty one), and some authentication stacks need explicit configuration to honor the headers — verify on your cluster rather than assuming.

For agent-operated fleets this inverts the credential problem. Instead of issuing every agent its own long-lived kubeconfig per tenant — a secret-sprawl nightmare that multiplies with tenants times agents — you issue the agent platform one identity with tightly scoped impersonation rights, and every tool call executes as the tenant it names. The tenant's RBAC stays the enforcement point; nothing in the agent path bypasses it. When the tenant's access is revoked, the agent's power over that tenant vanishes with it, with no agent-side credential to rotate. That is a strictly better revocation story than any per-tenant token scheme, and it falls out of primitives that have been stable in Kubernetes for years.

What you get free from clientcmd, and where parity stops

Everything above is not an argument for reading the clientcmd source for fun. It is an argument for not reimplementing it. Any Go CLI — your deploy tool, your operator console, the process behind your MCP server — gets the entire contract by wiring cli-runtime's generic flags instead of inventing its own --cluster --user --token trio:

go
configFlags := genericclioptions.NewConfigFlags(true)
configFlags.AddFlags(cmd.Flags())
// now your command speaks --kubeconfig, --context, --cluster,
// --user, --namespace, --server, --as, --as-group, and more,
// resolved through the same loading rules as kubectl

That small block buys three things no bespoke client offers. First, every operator and every agent harness already knows the auth model — contexts, kubeconfig chains, --as — so your tool is operable on day one by anyone who can drive kubectl, and scriptable by every existing runbook. Second, config-flag-based CLIs compose with the ecosystem for free: krew-style plugin conventions, KUBECONFIG-aware wrappers, and audit tooling all keep working because your binary quacks like kubectl at the auth layer. Third, the merge footguns from the previous section become shared, documented behavior with upstream fixes rather than your team's private bug inventory.

But kubectl parity is a foundation, not a product, and this is where the honest boundary goes. A Render-compatible deploy UX still needs its own verbs on top: deploy-from-git, build logs, service URLs, preview environments, per-service scaling knobs — none of that is in the kubectl vocabulary, and bolting it on as annotations is how platforms end up with a second, shadow API.

The same boundary applies to the MCP surface you put in front of agents. The ecosystem's hard-won lesson in 2026 — from read-only servers like kubernetes-mcp to production deployments fronting twenty-plus clusters, to governance backplanes like kagent's MCPServer CRD and policy-gated gateways — is to expose scoped tools (get-pods, tail-logs, rollout-status) with impersonation and auth can-i preflights baked in, never a raw run_kubectl(command: string) escape hatch that re-grants the model the full API behind a stringly-typed door. kubectl parity tells the agent which fleet it is on and whose rights it holds; your tool design decides what it may do there. Confuse the two layers and you get either an agent that cannot act or one that cannot be contained.

The through-line is simple. Kubeconfig discovery, merge precedence, and impersonation are solved problems with stable, upstream-maintained implementations — adopt them whole, footguns documented, instead of growing a parallel auth dialect your operators must learn and your agents will misread. Spend your originality budget one layer up, on the deploy verbs and the scoped agent tools that are actually your product. Your CLI should behave like kubectl right up to the exact line where your platform starts — and not one flag further.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources

  • Kubernetes documentation, "Organizing Cluster Access Using kubeconfig Files" — loading order and merge rules (kubernetes.io)
  • Kubernetes documentation, "Authenticating" — user impersonation headers and --as flags (kubernetes.io)
  • clientcmd package reference, NewDefaultClientConfigLoadingRules and config-flag plumbing (pkg.go.dev)
  • Testkube PR #8102 — single-file BuildConfigFromFlags versus multi-file KUBECONFIG chains (github.com)
  • "Build a Kubernetes MCP Server: Safe kubectl Access for AI Agents" — why scoped tools beat a raw command string (dev.to)
  • Prateek Raj, "How We Deployed a Kubernetes MCP Server on Production" — read-only MCP server across 20+ clusters with kagent (medium.com)
  • evoila/meho — policy-gated, MCP-native governance backplane for agents acting on infrastructure (github.com)

Related articles

Give your agents a chain backend

Autonomous agents hit RPC endpoints very differently than people do. See what bex router handles on their behalf.

Read the agents guide