Kubelet Config Drop-in Directories Just Went GA: What Kubernetes 1.35 Actually Buys a Cluster API Fleet

If you're still shipping one monolithic KubeletConfiguration file per node pool, Kubernetes 1.35 quietly removed the reason to. The --config-dir kubelet flag — a directory of small, numbered configuration snippets that get merged automatically at kubelet startup — graduated from beta to General Availability with the 1.35 release, after two full release cycles as alpha (v1.28) and beta (v1.30). It's stable, documented, and running in production clusters today.

Here's what that actually buys a fleet built on Cluster API: a GPU node pool's eviction thresholds, an edge node pool's resource reservations, and a security-hardening pass can each ship as their own file instead of forking the whole kubelet config three ways. Below is the exact KubeadmConfigTemplate diff that does it, and — because the TODO that inspired this post asked the harder question — the concrete drift risk that shipping smaller files does not make go away.

What --config-dir actually merges, and how​

The mechanics are simple by design. Point kubelet at a directory instead of (or in addition to) a single --config file, drop KubeletConfiguration fragments into it, and kubelet merges them at startup. The community convention — not an enforced schema, just what the GA announcement and most real-world examples use — is a numeric prefix that doubles as merge order:

# /etc/kubernetes/kubelet/config.d/00-base.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDNS: ["10.96.0.10"]
maxPods: 110
 
# /etc/kubernetes/kubelet/config.d/50-gpu-nodes.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
systemReserved:
  cpu: "1"
  memory: "4Gi"

Files merge in alphanumeric order, and later files win on any field they both set — 50-gpu-nodes.yaml overrides 00-base.yaml wherever they overlap, everything else from the base file passes through untouched. That's the whole mechanism: no templating engine, no external tool, no full-file diff to review when only one field changed.

Two operational details matter more than the merge logic itself:

• Validation happens at kubelet startup. A bad merge doesn't corrupt a running node silently — kubelet refuses to start and logs the validation error. You find out at boot, not at 3am when a pod gets evicted under a threshold nobody remembers setting.
• The merged result is inspectable, not just inferable. kubectl proxy plus a GET against /api/v1/nodes/<node>/proxy/configz returns the fully-merged KubeletConfiguration kubelet is actually running with — the authoritative answer to "what did these three files add up to on this specific node," independent of what the files on disk claim.

That second point is going to matter again in a few paragraphs, once we get to what this feature doesn't fix.

The old way vs. the new way on a Cluster API fleet​

Cluster API's KubeadmConfigTemplate already has a documented pattern for shipping a full KubeletConfiguration: write it to disk via spec.files, then point kubelet at it with kubeletExtraArgs.

# Before — one file, every field, every node pool that uses it
kubeadmConfigSpec:
  files:
    - path: /etc/kubernetes/kubelet/config.yaml
      owner: "root:root"
      permissions: "0644"
      content: |
        apiVersion: kubelet.config.k8s.io/v1beta1
        kind: KubeletConfiguration
        maxPods: 110
        kubeReserved: {cpu: "500m", memory: "1Gi"}
        systemReserved: {cpu: "250m", memory: "512Mi"}
        evictionHard: {memory.available: "500Mi", nodefs.available: "10%"}
  initConfiguration:
    nodeRegistration:
      kubeletExtraArgs:
        - name: config
          value: /etc/kubernetes/kubelet/config.yaml

The Cluster API Book is upfront about the tradeoff: this approach "is easy to replace the whole kubelet configuration generated by kubeadm, but it is not easy to replace only a part of it." Every MachineDeployment that needs one different field needs its own complete copy of every other field too, or a templating layer bolted on top to stamp out near-duplicate files. On a fleet with three or four node-pool variants, that's three or four nearly-identical 20-line files where the actual differences are two lines each.

Swap config for config-dir and split the same content into a shared base plus per-pool overrides:

# After — a shared base, one small file per pool that actually differs
kubeadmConfigSpec:
  files:
    - path: /etc/kubernetes/kubelet/config.d/00-base.yaml
      content: |
        apiVersion: kubelet.config.k8s.io/v1beta1
        kind: KubeletConfiguration
        maxPods: 110
        kubeReserved: {cpu: "500m", memory: "1Gi"}
    - path: /etc/kubernetes/kubelet/config.d/50-gpu-eviction.yaml
      content: |
        apiVersion: kubelet.config.k8s.io/v1beta1
        kind: KubeletConfiguration
        evictionHard: {memory.available: "1Gi", nodefs.available: "15%"}
  initConfiguration:
    nodeRegistration:
      kubeletExtraArgs:
        - name: config-dir
          value: /etc/kubernetes/kubelet/config.d

Worth being precise here, because Cluster API already ships a different layered-merge mechanism that's easy to conflate with this one: kubeadm's own patch files (kubeletconfiguration0+strategic.json, applied in alphanumeric order by suffix). Those patches are consumed once, by kubeadm, at init/join time, to modify the config kubeadm is about to generate — a one-shot transform. --config-dir is different in kind: kubelet itself reads whatever files currently sit in that directory, every time it starts, with no kubeadm step in between. Same "numbered files, alphanumeric merge" shape, two different consumers, two different lifecycles. Don't reach for kubeadm patches to solve a problem --config-dir already solves more directly, and don't assume the reverse either.

What this buys a fleet that's always adding machines​

The payoff isn't the syntax — it's what stops requiring a full-file change. On a Cluster-API-based platform provisioning fresh machines continuously (a new GPU pool this month, an ARM edge pool next month, a security-hardening pass rolled out incrementally after that), each of those used to mean either a new node image variant baked and tested end-to-end, or a new complete KubeletConfiguration file duplicating everything the base already had.

With --config-dir, each of those is one small file: 50-gpu-eviction.yaml, 60-arm-reserved.yaml, 90-security-hardening.yaml. The base config ships once, baked into the shared node image or the common KubeadmConfigTemplate fragment every pool inherits from. A tuning change to eviction thresholds for one pool touches one file with two fields in it — not a 20-line file that happens to duplicate 18 lines from every other pool's copy, and not a node-image rebuild pipeline run for a two-field change.

That's a real reduction in blast radius per change. It is not, on its own, a solution to keeping a growing, heterogeneous fleet consistent — which is the part the feature's own GA announcement doesn't dwell on and a Cluster-API operator needs to plan for directly.

The drift risk --config-dir doesn't solve​

Smaller files are easier to review and easier to get right individually. They do not change how those files reach a node, and that's exactly where Cluster API's own architecture reintroduces the drift the feature is marketed as solving.

KubeadmConfigTemplate is consumed once, at Machine creation. Kubeadm renders and writes those files when a node joins the cluster — not continuously, not on a watch loop. Edit a KubeadmConfigTemplate to add 50-gpu-eviction.yaml or change a value inside it, and every Machine that already exists keeps exactly the file set it joined with. Nothing about --config-dir retroactively pushes the new snippet to running nodes; that was never --config-dir's job, and Cluster API doesn't do it either. Getting the change onto existing nodes still requires the same lever it always has: a MachineDeployment rolling replace.

Kubelet doesn't hot-reload the directory. Even a node that somehow gets a new file written to /etc/kubernetes/kubelet/config.d/ after boot — an SSH command, a config-management run — won't act on it until kubelet restarts. A file being present on disk and a setting being active in the running kubelet are two different facts, and only one of them is visible from ls.

Put those two together and a fleet mid-rollout can genuinely be in three states at once, none of them visible from kubectl get machines:

The only way to tell which bucket a given node is actually in is the same /configz endpoint from the mechanics section — not the template diff, not the file listing, the live merged config kubelet reports for that one node. "The KubeadmConfigTemplate says X" and "the fleet is running X" are different claims, and drop-in files don't close that gap; they just make each individual file small enough that the gap is easy to forget about.

For a Cluster-API-based platform, that argues for treating a drop-in-file change exactly like any other kubeadm-config change, not as a lighter-weight exception to normal rollout discipline:

1. Version every snippet in git next to the KubeadmConfigTemplate it belongs to. Never hand-edit a file on a live node — that node's state stops matching source control the moment you do, and nothing will tell you it happened.
2. Force a full MachineDeployment rollout for every drop-in change, not an in-place file push to existing machines. A rolling replace is slower than editing a file over SSH; it's also the only path that keeps "template updated" and "node updated" the same event instead of two events with an unbounded gap between them.
3. Sample /configz across the rollout before calling it done. The template diff tells you what should be true. Only the running kubelet's own reported config tells you what is true on a given node, and that's the number worth checking before signing off on a kubelet-tuning change to production capacity.

None of this is a knock on the feature — GA drop-in directories are a real improvement over hand-maintaining one monolithic file per node variant, and the smaller diffs are easier to review precisely because they're smaller. The point is narrower: "composable" describes the file format, not the fleet's consistency guarantees. Those still come from whatever rollout discipline sits on top — in Cluster API's case, a MachineDeployment rolling replace and a habit of checking /configz, not the drop-in directory by itself.

Bex.co builds its fleet provisioning directly on Cluster API and Cluster API Provider Hetzner — the open-source, AI-native Render alternative: push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• Kubernetes v1.35: Kubelet Configuration Drop-in Directory Graduates to GA
• KEP-3983 — Add support for a drop-in kubelet configuration directory
• Cluster API Book — Kubelet Configuration in KubeadmConfig/KubeadmConfigTemplate
• Cluster API Provider Hetzner (CAPH) — GitHub
• CAPH Docs — Custom Node Images for Nodes in Clusters Managed by CAPH