Skip to main content

KEDA v2.20's One-Line RBAC Change Silenced Your Autoscaler's Events for 60 Days

11 min readDora NodaDora Noda
Share
On this page

You upgraded KEDA to v2.20, you read the upgrade note, you did the responsible thing and granted the operator create/patch on events.k8s.io/events. Then your keda-operator log filled up with this:

text
event.go:359 "Server rejected event (will not retry!)" err="events is forbidden:
User \"system:serviceaccount:keda:keda-operator\" cannot create resource \"events\"
in API group \"\" in the namespace \"keda\"" event=...

API group "". The old one. The one the release note said you were migrating away from.

This is not your RBAC being wrong. It is KEDA's own shipped manifests being wrong, for 60 days, on the default install path — and the failure mode is the nastiest one an autoscaler has: everything keeps scaling, and the record of why it scaled quietly stops existing.

Here is the fix first, then who actually needs to do anything, then the mechanism, then the rest of the v2.20.x minefield.


The fix, up front

The events rule in KEDA's operator ClusterRole has been in three different states in three releases:

KEDA versionReleasedevents rule in config/rbac/role.yamlResult
v2.19.02026-02-02apiGroups: [""]Events work
v2.20.02026-06-01apiGroups: ["events.k8s.io"]All event recording forbidden
v2.20.12026-06-08apiGroups: ["events.k8s.io"]All event recording forbidden
v2.20.22026-07-31apiGroups: ["", "events.k8s.io"]Events work

The whole regression is one kubebuilder marker in controllers/keda/scaledobject_controller.go. In v2.20.0 and v2.20.1:

go
// +kubebuilder:rbac:groups="events.k8s.io",resources=events,verbs=create;patch

In v2.20.2:

go
// +kubebuilder:rbac:groups="";events.k8s.io,resources=events,verbs=create;patch

That's it. That's the whole bug. The correct rule, in YAML, is:

yaml
- apiGroups: ["", "events.k8s.io"]
  resources: ["events"]
  verbs: ["create", "patch"]

Both groups. Never one.

Do you actually have to do anything?

Mostly no — which is exactly the point most upgrade guides get wrong. The RBAC advice in the v2.20.0 release note ("grant events.k8s.io before upgrading") is only correct for people whose ClusterRole is not rendered from KEDA's own artifacts. Everyone else should just skip to v2.20.2 and let the shipped manifest fix it.

Install methodOn v2.20.0 / v2.20.1Action to fix
kubectl apply -f keda-2.20.x.yaml (kustomize release artifact)Broken — the kubebuilder marker is the source of truth and it dropped ""Apply the v2.20.2 artifact; no manual RBAC
Helm chart kedacore/keda 2.20.0 / 2.20.1Broken — chart rendered apiGroups: [events.k8s.io] onlyhelm upgrade to chart 2.20.2 (2026-07-31); no manual RBAC
OLM (keda-olm-operator)Not affected yet — latest release is v2.19.0 (2026-04-22); the core-group grant landed in main on 2026-07-20, ahead of any 2.20 CSVNothing; upgrade when the 2.20 CSV ships
Hand-maintained / forked ClusterRole, OPA-or-Kyverno-constrained RBAC, or any policy that pins allowed apiGroupsBroken, and stays broken after upgrading to v2.20.2Add "" back to your own rule — this is the only group that must act

The exposure window on both self-service channels is the same: 2026-06-01 → 2026-07-31, 60 days. If you upgraded KEDA any time in June or July from a chart or the release YAML, you have been running without events since.

Already upgraded? Triage in two commands

kubectl get clusterrole is the obvious check and the wrong one — it reads a single object and misses aggregation and extra bindings. Ask the API server what the service account can actually do, in both forms:

bash
# core "" group — the one v2.20.0/.1 dropped
kubectl auth can-i create events \
  --as=system:serviceaccount:keda:keda-operator -n keda
 
# events.k8s.io group
kubectl auth can-i create events.events.k8s.io \
  --as=system:serviceaccount:keda:keda-operator -n keda

Both must print yes. Then confirm from the operator's own mouth:

bash
kubectl -n keda logs deploy/keda-operator --since=1h | grep -c "Server rejected event"

Anything above zero means every KEDA event in that window was dropped on the floor.

Making a manual fix survive your GitOps controller

If you do belong in that last row, do not kubectl edit the ClusterRole. Helm's next upgrade re-renders it, Argo CD's self-heal reverts it, and OLM's CSV reconcile overwrites it — the patch disappears at the least convenient moment, usually weeks later, with no log line saying so.

Ship a separate, additive object you own instead. It is invisible to KEDA's own renderer, so nothing reconciles it away:

yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: keda-operator-core-events
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]
yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: keda-operator-core-events
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: keda-operator-core-events
subjects:
  - kind: ServiceAccount
    name: keda-operator
    namespace: keda

This is the workaround the original bug report confirmed: events start landing again immediately, no restart required.


Why granting the new group alone is never enough

The trap is five years old and has nothing to do with KEDA. It is kubernetes/kubernetes#94857: client-go's event broadcaster writes to the legacy core "" API group even when it is configured for events.k8s.io. The new API group is a presentation of events, not a replacement storage path. Grant only the new group and you get exactly the error at the top of this post — with a message that names group "", which reads like your config is stale when it's actually your config being too new.

KEDA's migration was made in good faith. PR #7781 bumped the Kubernetes dependencies to 0.35.5 and switched the event emitter to the events.k8s.io recorder, then ran make manifests. The regenerated RBAC faithfully reflected the new recorder and silently dropped a rule that had been in the tree since 2020. As the fix PR on the Helm side put it: the event recorder was migrated, "however, internal dependencies were not searched."

Two of those dependencies matter:

  • client-go's broadcaster dual-write, above.
  • controller-runtime's leader election, which emits plain core events on every lease acquisition and is completely independent of whatever recorder KEDA chose.

The operational rule that survives this specific incident: on any controller upgrade that mentions events, events.k8s.io is additive, never a substitution. If a diff removes apiGroups: [""] from an events rule, that diff is wrong.


What you actually lose — and what you don't

Let's settle the blast radius flatly, because it decides your urgency: scaling keeps working. HPAs still receive metrics, ScaledObjects still activate and deactivate, scale-from-zero still fires. The original report classifies the impact as cosmetic: only Kubernetes Event objects are lost.

"Cosmetic" is fair for a single app. For a platform where every tenant's scale-from-zero path runs through one controller, it is the wrong word, because of what those events are:

EventEmitted whenWhat it's the only record of
KEDAScaleTargetActivatedTarget scaled 0 → 1 by a scalerWhich trigger woke a sleeping app
KEDAScaleTargetDeactivatedTarget scaled to 0That the platform chose to sleep it
KEDAScaleTargetActivationFailedScale to 1 failedA cold start that never happened
KEDAScalersStarted / KEDAScalersStoppedScaler watch loop lifecycleThat the trigger is even being polled
KEDAScalerFailedScaler can't reach its event sourceBroken queue creds, wrong endpoint
ScaledObjectCheckFailedValidation failedA tenant shipped an invalid trigger

Metrics tell you the replica count. Events tell you why the number changed. Delete them and a scale-from-zero path has no narrative record at all — and the two states that matter most, "nothing is scaling" and "everything is scaling and I have no idea why," produce identical evidence: an empty kubectl describe scaledobject.

That is what makes an autoscaler the worst place in the stack to hide an RBAC regression. A silenced webhook throws. A silenced ingress 502s. A silenced autoscaler looks exactly like a healthy autoscaler with nothing to do — until a tenant asks why their worker slept through a full queue, and your entire forensic trail is a replica count and a shrug.


The rest of the v2.20.x minefield

The events change also broke an event that was granted, for a subtler reason. The events.k8s.io recorder deduplicates on a different aggregation key than the legacy one, so the ScaledJob KEDAScalersStarted ("Started scalers watch") event collided with the per-scaler "scaler is built" event and vanished. v2.20.1 fixes it by setting a unique event action — a reminder that migrating event APIs changes dedup semantics, not just the endpoint.

Then the panics, all fixed after .0:

  • v2.20.1 — a concurrent map read/write data race in the fallback updateStatus path that panicked the operator when multiple triggers scaled simultaneously. Exactly the load profile of a multi-tenant fleet.
  • v2.20.2 — concurrent map writes in the shared root CA CertPool; a nil-pointer in GetCurrentReplicas when the informer cache returns a Deployment with an undefaulted spec.replicas (now treated as the Kubernetes default of 1); a nil-pointer guard on Status.ScaleTargetGVKR; and a nil-pointer in customScalingStrategy.GetEffectiveMaxScale when customScalingQueueLengthDeduction is omitted.

Add a restored gRPC reconnect backoff in v2.20.2 — an unset Backoff in WithConnectParams had disabled backoff entirely, producing a zero-delay reconnect loop that flooded logs whenever keda-operator was unreachable — and the conclusion writes itself: v2.20.0 and v2.20.1 are not fleet-grade. Go straight to v2.20.2.


HPAActive: the condition that un-breaks GitOps

v2.20.0 shipped a genuinely good fix that immediately caused a second problem. Issue #7649 — "ScaledObject Ready condition not reflecting HPA status" — was real: a ScaledObject could report Ready=True while its HPA showed <unknown> targets and ScalingActive=False. Healthy-looking object, silently broken scaling.

So v2.20.0 folded HPA status into Ready. And then every rolling restart started failing GitOps health gates: freshly-created pods have no metrics yet, the HPA briefly reports HPAMetricsUnavailable, Ready flips to False, and Argo CD's built-in keda.sh/ScaledObject health check — or your argocd app wait — declares the app unhealthy during a completely normal deploy.

v2.20.2 separates the two concerns with a dedicated HPAActive condition:

  • Ready — ScaledObject-level validity only. Trigger config parses, auth resolves, the object is well-formed. Safe for a GitOps controller to gate on again.
  • HPAActive — mirrors the HPA's own ScalingActive, with its real reasons (HPAMetricsUnavailable, ScalingDisabled). This is your alerting signal, and it's the one that answers "is scaling actually happening right now?"

For a platform team the split is the useful part: Ready belongs in your deploy gate, HPAActive belongs on a dashboard with a "flapping for > 5 minutes" alert. Before v2.20.2 you had one condition trying to be both, and it was wrong for at least one of them.


Pre-upgrade checklist

For a fleet where every tenant's scale-from-zero runs through one controller:

  1. Skip .0 and .1. Target v2.20.2 (KEDA and chart, both released 2026-07-31) or later. Nothing in .0/.1 is worth two known panic classes.
  2. Check Kubernetes first. KEDA 2.20 supports Kubernetes v1.33–v1.35. Upgrade the cluster before the autoscaler, never the reverse.
  3. Inventory who renders your RBAC. If the answer is "the chart" or "the release YAML," you have no manual work. If it's a fork, a Kyverno/OPA apiGroup allowlist, or a hand-written ClusterRole, add "" to the events rule before you upgrade.
  4. Grant both groups, alwaysapiGroups: ["", "events.k8s.io"] — and if you patch, patch with an additive ClusterRole + binding you own, so Helm/Argo/OLM can't revert it.
  5. Clear the deprecations. v2.20 removes, not deprecates: GCP PubSub subscriptionSize (use mode + value), InfluxDB authToken in triggerMetadata (move to authParams/resolvedEnv), Huawei Cloudeye minMetricValue, IBM MQ tls. Grep every tenant ScaledObject before the rollout, not after.
  6. Verify post-upgrade in this order: kubectl auth can-i for both event groups → grep -c "Server rejected event" in the operator log (must be 0) → force one scale-from-zero on a canary tenant and confirm KEDAScaleTargetActivated shows up in kubectl describe scaledobject. An event you have seen arrive is the only proof the RBAC is right.
  7. Re-point your alerts at HPAActive. Move HPA-health alerting off Ready, and re-enable any GitOps health check you disabled during the v2.20.0/.1 flapping.

Then add the one check that would have caught this whole class of bug in June: alert on the absence of KEDA events. If a fleet that normally emits a few hundred KEDAScaleTarget* events an hour emits zero for thirty minutes, that is either a total scaling outage or a silenced controller. Both are worth a page, and neither shows up in a replica-count graph.


KEDA's handling of this was, honestly, exemplary — the regression was found, reported with a reproduction, fixed on the chart, the OLM operator, and the source-of-truth kubebuilder marker within weeks. The lesson isn't "KEDA broke something." It's that a one-line change in generated RBAC can delete a component's entire observability surface while every dashboard stays green, and that scale-to-zero is precisely where you cannot afford that.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Scale-to-zero on hardware you control means the autoscaler's audit trail is yours too. Star the repo on GitHub.


Sources

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex