Kubernetes 1.36 Made Mutating Webhooks Optional: What Your Admission Layer Can Delete — and What Has to Stay

Every pod your tenants create currently takes a detour through infrastructure you operate: a mutating webhook Deployment with its own Service, its own TLS certificates, its own scaling story — sitting in the hot path of every single API-server write it touches. If that webhook is down, slow, or serving an expired certificate, pod creation across your fleet stalls or silently skips the defaults your platform promised.

As of Kubernetes 1.36, a large class of that machinery is deletable. MutatingAdmissionPolicy is now stable and enabled by default, which means CEL expressions evaluated inside the API server can set defaults, inject sidecars, and rewrite fields — no HTTPS callback, no separate service, no failurePolicy dilemma. Here is the verdict up front: default CPU/memory requests and limits, baseline NetworkPolicy fields, and sidecar injection — the three tenant-defaulting jobs a multi-tenant platform runs on nearly every pod — all move into declarative CEL policy. What stays behind a webhook is anything that needs to look at the world outside the request: image-signature lookups, external quota databases, generated secrets. The rest of this post is the migration map, the honest boundary line, and the upgrade checklist.

What actually shipped in 1.36​

Kubernetes 1.36 ("Haru"), released April 22, 2026, graduated MutatingAdmissionPolicy to general availability under KEP-3962. The fact box that matters for a fleet operator:

• Stable API, on by default. admissionregistration.k8s.io/v1 needs no feature gate on 1.36+. If you enabled the beta on 1.34/1.35 with --feature-gates=MutatingAdmissionPolicy=true plus the v1beta1 runtime config, both flags are now obsolete — Talos 1.13's upgrade notes list removing them as an explicit step.
• Policy/binding split. A MutatingAdmissionPolicy defines what changes; a MutatingAdmissionPolicyBinding decides where it applies (with paramRef supplying extra context like namespace labels). One policy, many bindings — the same shape ValidatingAdmissionPolicy established when it went GA back in 1.30.
• Two patch strategies. ApplyConfiguration for declarative "ensure this subtree looks like this" mutations, JSONPatch for surgical operations. CEL expressions must evaluate to the corresponding object shape, which the API server type-checks.
• The obvious precedent. Validating admission went through exactly this arc: webhooks first, in-process CEL later, GA in 1.30. Mutation is simply the half that took six more releases to finish.

This closes the gap for every team that kept a webhook server alive only because CEL previously covered validation but not mutation.

The before: what a tenant-defaulting webhook actually costs​

To feel what "deletable" is worth, inventory what a single hand-rolled mutating webhook — say, the one defaulting tenant pod specs — demands from a small platform team.

A service in the admission hot path. The webhook is a Deployment plus Service that every matching API write blocks on, governed by timeoutSeconds (max 30s) and your choice of failurePolicy. That choice is a genuine dilemma, and both sides have production scars. Fail means a webhook outage becomes a cluster-wide creation outage: one operator's failurePolicy: Fail default was flagged as "any webhook outage is a cluster-wide pod-creation outage," and another project hit exactly that when a startup certificate race left the webhook down and pod creation blocked behind it. Ignore fails open instead — pods start without your defaults, silently defeating whatever guarantee the webhook existed to provide.

There is no third setting; you pick which failure mode you prefer to be paged for.

TLS lifecycle for a component nobody asked for. The API server will only talk to your webhook over TLS with a CA bundle it trusts, so you run cert-manager (or equivalent) issuing and rotating serving certificates, with the CA injected into the webhook configuration. Certificate rotation is automatable right up until the automation breaks — the startup cert race above is a real-world example — and then admission for the whole fleet is down because of a certificate on a helper service.

Per-request latency on every write. One published comparison puts external-webhook admission at a 5–100ms network round trip versus sub-millisecond in-process evaluation for native CEL policies. Treat the exact numbers as order-of-magnitude, but the direction is structural: a webhook adds a network hop, TLS handshake, serialization round trip, and your handler's own runtime to every admission decision. CEL runs inside the API server process.

A scaling and bootstrap problem. The webhook must scale with API-server write load, needs PodDisruptionBudgets and monitoring of its own, and creates a chicken-and-egg at cluster bootstrap: the thing that configures workloads must itself be deployed, healthy, and trusted before the workloads it configures can start. On a Cluster API-managed fleet, where machines and control planes are themselves declaratively provisioned, this is the one component that resists the declarative model — it is imperative infrastructure underneath your declarative platform.

The after: three migrations, concretely​

Each migration below follows the same shape: the webhook you run today, the CEL policy that replaces it. Sketches, not copy-paste manifests — adapt names and match criteria to your fleet.

1. Default CPU/memory requests and limits (the quota-adjacent job)​

The most common tenant-defaulting webhook in a multi-tenant fleet ensures no container runs without resource requests and limits — the input your bin-packing, per-tenant accounting, and noisy-neighbor protection all assume exists. Today that is a webhook (or a LimitRange plus a webhook for tenant-specific defaults); with 1.36 it is a policy:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingAdmissionPolicy
metadata:
  name: default-tenant-resources
spec:
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE"]
      resources: ["pods"]
  mutations:
  - patchType: ApplyConfiguration
    expression: |
      Object{
        spec: Object.spec{
          containers: object.spec.containers.map(c,
            has(c.resources) && has(c.resources.requests) ? c :
            c + {'resources': {'requests': {'cpu': '100m', 'memory': '128Mi'}}})
        }
      }

The real expression merges per-container defaults (e.g. requests: {cpu: "100m", memory: "128Mi"}) only where the tenant did not set them; the sketch above shows the structure — match pods at CREATE, map over containers, fill in what is missing. Its simpler cousin deserves a passing mention: default labels (managed-by, tenant id) are the same pattern with a smaller object, and make a good first migration to build confidence.

One honest boundary inside this very example: defaulting a container's requests/limits is admission mutation, but ResourceQuota objects and quota accounting are separate built-ins. CEL sets the fields; it does not create quota objects or track usage. If your "quota webhook" was actually enforcing budgets against a database, that half stays a webhook (see the boundary section).

2. Baseline fields on tenant-authored NetworkPolicies​

The second job: whenever a tenant creates a NetworkPolicy, ensure it carries your baseline — say, an explicit policyTypes including Ingress so nothing accidentally deploys wide open. Same pattern, different resource:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingAdmissionPolicy
metadata:
  name: baseline-tenant-netpol
spec:
  matchConstraints:
    resourceRules:
    - apiGroups: ["networking.k8s.io"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["networkpolicies"]
  mutations:
  - patchType: ApplyConfiguration
    expression: |
      Object{
        spec: Object.spec{
          policyTypes: has(object.spec.policyTypes)
            ? object.spec.policyTypes : ["Ingress"]
        }
      }

Note what this does not do: provision a default-deny policy into each new tenant namespace. Creating objects that do not exist yet is a controller's job (watch namespaces, create the policy), not an admission mutation of an incoming request. That provisioning controller stays.

3. Sidecar injection​

The third classic webhook — inject the logging/metrics/proxy sidecar into every tenant pod — is directly expressible: match pods, append a container definition to the pod spec via the mutation expression. The shape is the same as migration 1, with a container appended rather than fields defaulted.

The practical limit: CEL is an expression language, not a templating engine. Static sidecar definitions migrate cleanly; sidecars requiring per-pod string surgery (generated names, computed arguments assembled from several fields) get unreadable fast, and that unreadability is a signal to keep the webhook.

What you gain, in one table​

The theme across every row: the admission layer stops being software you run and becomes configuration you declare — which is the entire premise a Cluster API fleet is built on.

Where the webhook stays: the boundary rule​

CEL inside the API server sees the request payload plus whatever the binding wires in as parameters (namespace labels, a referenced config object). It cannot call out to anything: no OCI registry lookup, no quota database, no secret generator. That draws the line crisply:

Stays a webhook. Verifying an image signature against a registry at admit time (the check needs the registry's answer, which is not in the request). Enforcing a budget held in an external system. Generating credentials or certificates per workload. Anything where the correct mutation depends on state the API server does not already have.

Moves to CEL. Anything decidable from the object itself plus bound params: defaults, baseline fields, label/tag stamping, static sidecars, namespace-scoped variations via paramRef (e.g. stricter defaults for the tier: free namespaces, roomier ones for tier: pro — one policy, two bindings).

The decision rule for auditing your existing webhooks: could this handler be rewritten as a pure function of the incoming object and a config map? If yes, it is a CEL policy now. If the handler's first step is a network call, keep the service — and consider shrinking it, since the defaulting half of its job just moved out.

Fleet-operator checklist​

1. Drop the beta flags. On 1.36+, remove --feature-gates=MutatingAdmissionPolicy=true and the admissionregistration.k8s.io/v1beta1 runtime config from API-server args; confirm kubectl api-resources --api-group=admissionregistration.k8s.io shows the stable resources.
2. Audit webhooks for CEL-portability. List every MutatingWebhookConfiguration in the fleet, apply the pure-function test above, and migrate the portable ones starting with label/default stamping (lowest blast radius).
3. Keep policy engines for the complex middle. Kyverno and Gatekeeper still own multi-object logic, generated resources, and anything beyond CEL's cost limits — the point is deleting hand-rolled webhook Deployments, not the policy layer.
4. Re-check the failure modes you delete. Each migrated webhook removes a TLS rotation, a failurePolicy choice, and a bootstrap dependency. Say so in the commit message; future-you will want the record of which outage class just closed.
5. Mind the forward direction. Validation walked this exact path (webhooks → CEL → GA in 1.30) and the ecosystem followed; expect tooling, examples, and policy libraries for the mutating side to compound the same way now that the API is stable.

The admission layer becomes config​

The deeper story is not one API graduation. For years, every Kubernetes platform carried a paradox: the control plane was declarative, but the component shaping every object entering it was imperative code you deployed, scaled, and nursed through certificate renewals. Kubernetes 1.36 ends that split for the large, boring, universal middle of admission — defaults, baselines, static injection — and names the remainder honestly: if your admission decision needs the outside world, run a service; if it does not, write a policy.

For a self-hosted PaaS on owned hardware, that trade lands especially well. You already chose to own the metal to escape per-request meters; now you can delete per-request infrastructure too — one fewer Deployment standing between your tenants and a running pod.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.