Field Is Immutable: Why helm upgrade Dies at the API Server, and Which Escape Hatch Actually Scales

The OpenTelemetry Collector chart shipped a release whose upgrade instructions are, in full: don't.

"We broke the selector labels in 0.110.0, which causes helm upgrades to fail. Do not attempt to upgrade from 0.110.0 to either 0.110.1 or 0.110.2. Go straight to 0.110.3 instead." — opentelemetry-collector UPGRADING.md

Nobody did anything wrong. A maintainer adjusted a label template; Kubernetes decided that was a different Deployment; helm upgrade returned a paragraph naming a field and offering no fix. This is not an OpenTelemetry story — it's a category. bitnami/minio broke standalone installs merging its standalone and distributed templates. bitnami/clickhouse broke on StatefulSet fields. bitnami/postgresql broke on clusterIP. The oldest Helm issue on the pattern, helm#2149, was opened in 2017 and the failure mode is unchanged.

CNCF's January 2026 platform-engineering maintenance post measured the surface this happens on: 14 major open-source projects in a typical internal platform produce 276–327 patches, 43–52 minor upgrades, and 2–5 major upgrades per year — roughly one update a day. Immutable-field breakage is a small fraction of those. But a fraction of one-a-day, multiplied across every cluster in a fleet, is a recurring outage-shaped tax, and the CNCF authors' recommendation is blunt: build a platform operator that manages changes to immutable Kubernetes properties, rather than a runbook.

Here is the taxonomy, the five things you can actually do about it, and where the operator advice stops being right.

The taxonomy: what's immutable, and the exact string you'll paste into a search box​

First, a correction that matters practically: this is not admission. Immutable-field rejection happens in each resource's ValidateUpdate strategy inside the API server's registry — before any webhook has an opinion. You cannot exempt it with a ValidatingWebhookConfiguration namespace selector, and no --force flag on any client reaches it.

Two notes on that table. Service type transitions are legal — ClusterIP ↔ NodePort ↔ LoadBalancer have worked for years. What bites is clusterIP itself, usually because a chart stopped emitting the field and Helm sent "". And the StatefulSet row is the reason KEP-0661 (StatefulSet volume resize) has been in flight for years without landing: growing a volumeClaimTemplates request is still not a supported in-place operation.

The lookalike that isn't this. If your error says Apply failed with 1 conflict: conflict with "helm", that is a server-side-apply field-manager dispute, not immutability. It is fixable with --force-conflicts, and applying that reflex to a real immutable field will do nothing except waste an hour.

The adjacent trap. Helm never upgrades CRDs in a chart's crds/ directory — they are installed on first helm install and ignored forever after, an explicit project decision made over data-loss risk. So a chart bump can silently leave your CRDs a version behind while the immutable-field error you're debugging is downstream of that.

Why --force isn't the answer​

helm upgrade --force swaps Helm's patch strategy for replace: it deletes the object and creates it again. That does get past ValidateUpdate, which is why every StackOverflow answer reaches for it, and it is the wrong instinct on exactly the resources where you hit the problem.

Replacing a Service releases its clusterIP and, for a LoadBalancer, frequently its external IP — the one in your DNS records. Replacing a Deployment skips the rolling update entirely: all old pods die, then new ones start. Replacing a StatefulSet with bound PVCs is the scenario the Helm maintainers' own thread has circled since 2020, and community reports are consistent that it does not reliably rescue StatefulSets anyway.

Worse, --force is indiscriminate. It applies to every resource in the release, not the one that failed. You reached for it to fix one label and restarted your whole stack.

Five paths, priced against each other​

The escape hatches are usually presented as a menu of techniques. They aren't comparable as techniques — they're comparable as costs, and the costs land in different places. Downtime is given per workload class, because that's the variable that actually swings the answer.

Path 1 is not a joke and it is often correct. Skipping a chart release costs nothing and resolves most of these incidents — the OpenTelemetry fix was literally "go to 0.110.3." Pinning stops being correct the moment the release you're skipping carries a CVE fix.

Path 4 disqualifies itself for the stated problem. Encoding a hash of the immutable spec into the resource name (redis-a3f9c1) turns every immutable change into a create-then-drain, which is elegant — and useless against a community chart, since you'd have to fork it. It also doesn't carry PVCs across the rename, needs a stable Service in front for DNS, and doubles capacity during cutover. Include it for completeness; don't plan around it.

The crossover is a multiplication, not a preference. Path 3's cost is O(clusters × affected charts) in careful human minutes, at 2am, on a stateful workload. Path 5's cost is a large fixed number followed by roughly zero. One team, three clusters, a dozen bundled charts: the runbook wins, comfortably, and building an operator is a way to avoid doing your job. Thirty clusters, or the same chart bump landing on every tenant of a platform you operate for other people: the runbook is now thirty chances to fat-finger a kubectl delete against production, and the operator's fixed cost amortizes on the first incident it handles unattended.

Path 3 in full, with the two gotchas​

The real sequence for a StatefulSet whose chart bumped volumeClaimTemplates:

# 1. Snapshot the live object. You will need it.
kubectl get sts pg -o yaml > pg-before.yaml
 
# 2. Grow the existing PVCs FIRST. Recreating the StatefulSet does not do this.
for i in 0 1 2; do
  kubectl patch pvc data-pg-$i -p \
    '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
done
 
# 3. Delete the controller, keep the pods and PVCs running.
kubectl delete sts pg --cascade=orphan
 
# 4. Recreate via Helm. Pods are adopted, not restarted.
helm upgrade pg oci://registry.example/charts/pg --version 17.0.2

Gotcha one: adoption is by selector, and it is silent when it fails. A new StatefulSet adopts orphaned pods only if its spec.selector still matches their labels. If the chart bump changed the selector — the most common trigger for the whole problem — the new controller sees zero pods, spins up a full replica set beside the orphans, and now two sets of processes are mounting the same storage. Diff the rendered selector against the live pod labels before step 3, not after.

Gotcha two: volumeClaimTemplates only governs future ordinals. Recreating the StatefulSet with 50Gi in the template does nothing to data-pg-0 through data-pg-2; those PVCs keep whatever they had. That is why step 2 comes first, and it only works if the StorageClass sets allowVolumeExpansion: true. If it doesn't, this path doesn't exist for you and you're doing a backup-and-restore migration instead.

Downtime is near zero for stateless and singleton workloads. For a quorum system — etcd, Kafka, ZooKeeper — the orphan window is when nothing is reconciling membership, so a node failure during it is unrecovered until you finish.

What the operator actually does​

The operator is not "automate the runbook." It's a classifier with a plan step. Concretely, for each release:

1. Render both versions. helm template at the pinned version and at the target, with the same values.
2. Diff by GVK + name, and classify each changed field against a fixed predicate table — the same knowledge as the taxonomy above, expressed as (kind, JSONPath) → verdict:

(Deployment,  spec.selector)            -> RECREATE
(DaemonSet,   spec.selector)            -> RECREATE
(StatefulSet, spec.volumeClaimTemplates)-> ORPHAN_ADOPT + PVC_EXPAND
(StatefulSet, spec.serviceName)         -> ORPHAN_ADOPT
(StatefulSet, spec.selector)            -> ORPHAN_ADOPT + SELECTOR_GUARD
(Service,     spec.clusterIP)           -> PRESERVE_LIVE_VALUE
(Job,         spec.template.*)          -> RECREATE
(Secret,      type)                     -> RECREATE
(*,           *)                        -> PATCH

3. Emit a plan — an ordered list of operations with a blast-radius label — and refuse to execute if any resource classifies as RECREATE on a workload marked stateful, unless a human approves.
4. Execute with health gates: verify adoption (.status.replicas equals live pod count), verify readiness, and roll back to the snapshot on failure.

Note PRESERVE_LIVE_VALUE on Service.spec.clusterIP: for the single most common instance of this error, the right answer isn't recreation at all. It's copying the live value into the rendered manifest so the field never changes.

Two costs the advocacy usually omits. The operator needs delete-and-create rights on workloads across every namespace it manages — a genuinely large RBAC grant, and now a component in your threat model. And a classifier that's wrong is worse than a runbook, because it's wrong at machine speed across the fleet. Ship it in dry-run-only mode for a quarter before it's allowed to touch anything.

Catch it at PR time, not at 2am​

The tempting one-liner does not work: Helm's own --dry-run=server does not surface immutable-field errors (helm#10869, helm#31790). It renders and it validates schemas; it does not rehearse the update against live objects. Neither does kubectl diff, which shows you a delta and says nothing about whether the API server will accept it.

What does work is routing the rendered manifests through an apply that the API server actually evaluates:

helm template myrelease chart/ --version "$NEW" -n prod \
  | kubectl apply --dry-run=server --server-side -f - -n prod

--dry-run=server runs the full update path — including ValidateUpdate — and discards the result, so an immutable delta fails here with the same string it would have produced in production. Run it in CI on the chart-bump PR against a cluster that has the workload; a non-zero exit means the bump needs a migration plan, not a merge. It's a few lines, it fails loudly, and it converts an incident into a code review.

The platform's problem, not the operator's​

Every self-hosted platform that bundles dependencies — an ingress controller, a Postgres operator, a metrics stack — inherits this. The tenant did not choose those charts and cannot be handed a runbook that begins "first, delete the StatefulSet by hand." A platform that ships a chart is committing to a migration path for it, which means the classification table above belongs in the platform's code, not in a wiki page someone reads under pressure.

That's also the honest reason to prefer a real operator over a Helm dependency for anything stateful — the CNCF post's other recommendation, and the reason projects like CloudNativePG exist. An operator knows what its resource is, so it can tell the difference between a field it may patch and a rebuild it has to sequence. Helm only knows what the YAML said.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• Platform engineering maintenance pitfalls and smart strategies to stay ahead — CNCF, January 21, 2026
• opentelemetry-collector chart UPGRADING.md — 0.110.x selector labels
• bitnami/charts#19763 — StatefulSet forbidden field update
• bitnami/charts#34103 — minio combined Deployment template breaks upgrades
• bitnami/charts#18215 — clickhouse upgrade broken by immutable fields
• bitnami/charts#3447 — postgresql immutable clusterIP
• helm#2149 — Error when updating StatefulSets (2017)
• helm#7998 — StatefulSet forbidden updates
• helm#10869 — Why isn't dry run catching this validation error?
• helm#31790 — --dry-run=server doesn't include SSA conflicts and other failures
• Helm docs — Custom Resource Definitions
• Kubernetes docs — Delete a StatefulSet
• KEP-0661 — StatefulSet volume resize