Chaos-Test Your Cluster API Fleet: Three LitmusChaos Experiments That Prove Self-Healing Before 3 A.M. Does

Your Cluster API fleet has a self-healing story it has never had to prove.

Somewhere in your management cluster a MachineHealthCheck says a dead node gets remediated automatically. It has a nodeStartupTimeout, a set of unhealthyConditions, and a maxUnhealthy: 50% you copied from an example. It has never fired in production because the one time a Hetzner worker went dark at 2:47 a.m., someone was awake, cordoned by hand, and replaced the Machine before the controller could decide. The incident ended with a Slack thread, not a receipt.

That gap is exactly what LitmusChaos's first-half 2026 update and Cluster API v1.14 closed. On August 6, 2026 the CNCF published LitmusChaos's Q1–Q2 2026 report — Juju charmed operators, an expanded fault catalog, and five fixes shipped upstream from Flipkart's central reliability team after running 90% of fault injection in staging. Two days earlier Cluster API v1.14.0-rc.1 shipped two fixes that had quietly blocked control-plane chaos for a year: forwarded etcd leadership before remediation and a guard against orphaned learner members stalling quorum checks. Together they mean you can break a CAPH or CAPD fleet on purpose and watch remediation converge — or fail — before a real node does it at 3 a.m.

This post is that suite. Three ChaosExperiment + ChaosEngine pairs with probes that prove the loop every platform page asserts: remediation fires within nodeStartupTimeout, workloads reschedule, and the fleet reaches Ready without a human.

Verdict up front. If you run Cluster API on Hetzner via CAPH — or on Docker via CAPD for staging — you can prove self-healing this week in one namespace with three faults. You need one trusted MachineHealthCheck and a checklist that treats remediation like a test assertion, not a runbook hope.

The suite at a glance​

Start here, then dive for context.

Each experiment runs 120–300 seconds, targets one Machine, and respects maxUnhealthy: 50% so a 3-control-plane / 2-worker fleet cannot lose quorum during the test. If any row fails, the loop is self-hoping. Fix the MachineHealthCheck, not the test.

The three experiments — copy-paste YAML with probes​

Install LitmusChaos once per cluster. Chaos runs in the workload cluster, observation happens in the management cluster where Machine objects reconcile.

# Workload cluster — where faults run
kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v3.0.0.yaml
kubectl create ns litmus
 
# Management cluster — where you watch remediation
kubectl get machines -A
kubectl get machinehealthchecks -A

Experiment 1: Kill the kubelet, watch the Machine die and come back​

The host is powered and the Hetzner API says running, but kubelet stopped reporting. nodeStartupTimeout covers "Machine without a Node," and unhealthyConditions covers Ready Unknown.

ChaosExperiment:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: worker-kubelet-kill
  namespace: litmus
  labels:
    name: worker-kubelet-kill
spec:
  definition:
    scope: Namespaced
    permissions:
      - apiGroups: [""]
        resources: ["pods","pods/exec"]
        verbs: ["create","delete","get","list","patch","update"]
    image: "litmuschaos/go-runner:3.0.0"
    imagePullPolicy: Always
    args: ["-c", "./experiments -name pod-delete"]
    command: ["/bin/bash"]
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "180"
      - name: CHAOS_INTERVAL
        value: "10"
      - name: FORCE
        value: "true"

ChaosEngine with probes — the actual assertions:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: worker-kubelet-kill-engine
  namespace: litmus
spec:
  appinfo:
    appns: kube-system
    applabel: "app.kubernetes.io/name=kubelet"
    appkind: daemonset
  chaosServiceAccount: litmus-admin
  experiments:
    - name: worker-kubelet-kill
      spec:
        probes:
          - name: remediation-fired
            type: cmdProbe
            mode: Continuous
            runProperties:
              probeTimeout: 5
              interval: 15
              retry: 40
            cmdProbe/inputs:
              command: |
                kubectl --kubeconfig=/etc/management/kubeconfig \
                  get machine -n default -o json \
                  | jq -e '.items[] | select(.status.conditions[]? | select(.type=="MachineOwnerRemediated" and .status=="True"))'
          - name: workload-rescheduled
            type: cmdProbe
            mode: Edge
            runProperties:
              probeTimeout: 5
              interval: 10
              retry: 30
            cmdProbe/inputs:
              command: |
                kubectl get pods -n default -l app=tenant-demo \
                  -o json | jq -e '[.items[] | select(.status.phase=="Running")] | length >= 2'

What success looks like:

• At T+0 the target Node flips Ready Unknown.
• Within nodeStartupTimeout the Machine shows HealthCheckSucceeded: False, then MachineOwnerRemediated: True with LastRemediated inside that window.
• The controller deletes the Machine — or reboots the HCloudMachine if you use remediationTemplate — and a new Machine reaches Running while the tenant Deployment runs on another worker.

If remediation never fires, check maxUnhealthy first. Two workers with maxUnhealthy: 50% allow one remediation; two simultaneous faults are short-circuited by design. Do not raise it to 100% to make the test pass.

Experiment 2: Partition one control-plane node — the test that used to be impossible​

Until v1.14 this was a paper exercise. Two bugs blocked it:

1. KCP did not forward etcd leadership before deleting a Machine. If the victim held the leader, reconcileEtcdMembers stalled.
2. A Machine whose Node never registered left an orphaned learner. The targetLearnerMembers > 0 guard in targetEtcdClusterHealthy then blocked any other voter remediation because the orphan stayed IsLearner=true.

Both are fixed in v1.14.0-rc.1 via #13848 and the learner-orphan cleanup around remediation.go:796-798. Testable now does not mean safe — it means quorum is finally observable.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: cp-network-partition
  namespace: litmus
spec:
  definition:
    scope: Namespaced
    permissions:
      - apiGroups: [""]
        resources: ["pods","pods/exec"]
        verbs: ["create","delete","get","list","patch","update"]
    image: "litmuschaos/go-runner:3.0.0"
    args: ["-c", "./experiments -name pod-network-loss"]
    command: ["/bin/bash"]
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "240"
      - name: NETWORK_PACKET_LOSS_PERCENTAGE
        value: "100"
      - name: DESTINATION_IPS
        value: "10.0.0.1" # peer control-plane IP — partition one edge only

Assertions while chaos is active:

kubectl get kubeadmcontrolplanes -A -o yaml | grep -A2 etcdLeaderCandidate
etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  member list -w table
# Expect: leadership moved before deletion, orphan learner count 0 after
kubectl get machines -A -o json | jq '.items[] | {name: .metadata.name, remediated: .status.conditions[]? | select(.type=="MachineOwnerRemediated")}'

Success is one remediation with leadership forwarded, not two. KCP's canSafelyRemediateMachine must reject concurrent control-plane remediations that would drop quorum — a gate still under discussion in the v1.14 cycle. Two simultaneous deletions means the guard failed, not the chaos.

Experiment 3: Fill the disk — the eviction fault teams forget​

Kubelet sets DiskPressure: True and the scheduler avoids the node, but the Machine stays Running until your MachineHealthCheck says pressure counts as unhealthy.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: disk-fill-80
  namespace: litmus
spec:
  definition:
    scope: Namespaced
    image: "litmuschaos/go-runner:3.0.0"
    args: ["-c", "./experiments -name disk-fill"]
    command: ["/bin/bash"]
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "180"
      - name: FILL_PERCENTAGE
        value: "80"
      - name: TARGET_PODS
        value: "tenant-demo-.*"
      - name: DATA_BLOCK_SIZE
        value: "1024"

Add the condition or this never remediates by design:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineHealthCheck
metadata:
  name: workers-disk-aware
  namespace: default
spec:
  clusterName: hetzner-fleet
  selector:
    matchLabels:
      node-role.kubernetes.io/worker: ""
  maxUnhealthy: "50%"
  nodeStartupTimeout: 10m
  unhealthyConditions:
    - type: Ready
      status: Unknown
      timeout: 5m
    - type: Ready
      status: "False"
      timeout: 5m
    - type: DiskPressure
      status: "True"
      timeout: 2m
  # CAPH path: uncomment to test reboot-vs-delete
  # remediationTemplate:
  #   apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  #   kind: HCloudRemediationTemplate
  #   name: hcloud-reboot

On CAPH, HCloudRemediationTemplate reboots, sets LastRemediated, bumps RetryCount, and enters Waiting. Success leaves no new server — assert the condition, not the server ID.

What MachineHealthCheck actually promises​

MachineHealthCheck is a remediation gate with four knobs. Each quietly changes what "self-healing" means:

CAPH nuance: MachineOwnerRemediatedCondition (condition_consts.go:167-169) marks that the infra provider finished. Bug #1983 showed the reboot path deleting the Machine even on reboot success — violating that contract. On v1.11+ templates verify the condition before you assert deletion.

Why August 2026 is the moment​

Two notes that read like trivia together unlock the suite.

LitmusChaos Q1–Q2 2026 (CNCF, August 6, 2026). The mid-year update headline is Flipkart's central reliability team winning the CNCF Case Study Contest for KubeCon India: a multi-tenant LitmusChaos platform across hundreds of microservices, 90% of fault injection in staging before festive traffic, five fixes contributed upstream. Plus Juju charmed operators for the control plane. The takeaway for a team that postponed chaos because "we need prod first": staging coverage already paid back upstream.

Cluster API v1.14 rc.0 (July 28) + rc.1 (August 4). The rc.1 notes carry the operator-relevant fixes: etcd leadership forwarding, learner-orphan guard, plus aggregated machine-version surfacing and tolerance for missing InfraTemplates during deletion — fewer "remediation failed because an unrelated object was missing" false negatives. CAPH v1.1.0 already requires CAPI v1.12+ and carries v1beta1 for its CRDs alongside v1beta2 core types, so a Hetzner fleet on supported versions consumes the new guards without a CRD flag day.

Machine remediation remains delete-and-recreate (or reboot-and-wait) with real cloud-API latency. Hetzner's rate limits and IPAM add minutes no annotation hides. The suite measures that latency, not pretends it is zero.

Running it on CAPH or CAPD without bricking the fleet​

Prerequisites​

• Management cluster on a separate failure domain from the workload cluster. Do not chaos-test the host of clusterctl state.
• Workload cluster with 3 control-plane Machines and at least 2 workers, or maxUnhealthy blocks everything and you learn nothing except that the guard works.
• Two MachineHealthChecks — the disk-aware one above plus a minimal control-plane MHC:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineHealthCheck
metadata:
  name: cp-health
  namespace: default
spec:
  clusterName: hetzner-fleet
  selector:
    matchLabels:
      cluster.x-k8s.io/control-plane: ""
  maxUnhealthy: "50%"
  nodeStartupTimeout: 10m
  unhealthyConditions:
    - type: Ready
      status: Unknown
      timeout: 3m
    - type: Ready
      status: "False"
      timeout: 3m

Execution order​

1. Deploy a canary workload with PodDisruptionBudget and two replicas. Assert this tenant, not kube-system.
2. Run in order: worker kubelet kill → disk fill → control-plane partition. Never stack.
3. Observe from the management cluster:

watch -n 2 'kubectl get machines -A -o wide; echo "---"; \
  kubectl get machines -A -o json | jq ".items[] | {name: .metadata.name, phase: .status.phase, remediated: (.status.conditions // [] | map(select(.type==\"MachineOwnerRemediated\")))}"'
kubectl logs -n capi-system deploy/capi-controller-manager -f | grep -i remediat
kubectl logs -n caph-system deploy/caph-controller-manager -f | grep -i remediat

4. Clean up: kubectl delete chaosengine -n litmus --all and kubectl delete chaosexperiment -n litmus --all. Litmus does not revert Machines — if the fleet is not whole, the suite found the bug.

Safety notes you would otherwise learn at 3 a.m.​

• Do not raise maxUnhealthy to make a failing test pass. A test needing 100% proves you will lose quorum in production.
• Control-plane chaos on a single-control-plane cluster is an outage, not a test.
• HCloud rate limits are part of remediation — assert within nodeStartupTimeout + 5m, not an exact second.
• With HCloudRemediationTemplate, a good reboot increments RetryCount but creates no new server.

Receipts, not promises​

Pass criteria (all must hold)​

What the suite does not prove​

• A full AZ or region loss — one pod-network partition is not a torn facility.
• Data-layer recovery — a remediated etcd member still needs a healthy snapshot path.
• Operator on-call load — someone still tunes unhealthyConditions for the workload you actually run.

The CNCF Flipkart case study won because it ran 90% of faults in staging and saw failures there first, not because prod became infallible. A quarterly run of these three — or a CI run on every infra-template change via CAPD — costs one namespace and ten minutes. An untested MachineHealthCheck costs a frozen fleet the morning after the management cluster's own bad night, when every scale and heal queues behind a brain no one proved could come back.

Run the faults. Keep the receipts. Market the receipts, not the CRD.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. The same Cluster API + CAPH fleet that runs these LitmusChaos experiments is the fleet Bex provisions declaratively: every Machine is an object, every remediation is a reconciler, and any agent can drive it through a Render-compatible API. Star the repo on GitHub or deploy your first app today.