Skip to main content

kpack: The Buildpack Controller That Rebuilds When the Base Layer Patches, Not When You Push

9 min readDora NodaDora Noda
Share
On this page

Paketo's own SLA for its buildpacks stacks promises a High or Critical CVE gets patched into the base image within 48 hours of the upstream fix landing. That's a good number. It's also completely irrelevant to your running app if the only thing that triggers a rebuild is git push — because the 48-hour patch sits in a registry doing nothing until you touch your code again, and on most teams' actual cadence, that's not 48 hours away.

This is the gap kpack was built to close. It's a Kubernetes-native implementation of Cloud Native Buildpacks — the same buildpacks lineage that powers Heroku, Google Cloud Run, and most git-push PaaS tools — but instead of treating a build as something that only happens in response to a source change, kpack treats the base image itself as a thing worth watching. When a new base image lands with a security patch baked in, kpack notices and rebuilds your app image on its own, with zero commits and zero developer action. "Rebuild on push" becomes "rebuild on push, or on Tuesday when the base image patches, whichever comes first."

The gap isn't hypothetical — Heroku already lived through it

If this sounds like an edge case, it isn't a new one. Heroku's own two build models drew this exact line years ago, and the difference is instructive. Apps on Heroku's classic buildpack model get their compiled "slug" laid on top of the current stack image every time a dyno restarts — so when Heroku patches a CVE into heroku-22, every buildpack-based app inherits it automatically, the next time it happens to restart, no redeploy required. Apps that instead build via Heroku's container/Docker path don't get that: the base image is baked into the image at build time, and picking up a patched base means rebuilding and redeploying the image yourself.

Same platform, same CVE, two completely different remediation timelines depending on which build path an app happened to use. That's the whole problem in miniature: whether a security patch reaches production depends on whether something is watching the base layer, or only watching your commits.

How rebuild-on-push actually works — and where the base image hides

Walk the standard flow on most git-push PaaS tools, bex included as it exists today: a push lands, the platform detects the language, buildpacks run detect → build → export, and the result gets pushed as a new OCI image and rolled out. The base image — the OS packages, the language runtime, the CA certificates — gets selected once at build time and baked into a layer of that image. It isn't a variable the pipeline revisits; it's a constant, chosen the moment the build ran and then frozen until the next build.

That's fine as long as builds keep happening. It stops being fine the moment an app goes quiet — no feature work, no bug fixes, just sitting there serving traffic. Nothing about a stable, well-running app makes it a candidate for a rebuild in a push-triggered pipeline. But it's exactly the kind of app most likely to be running an old base image, because nothing has forced a rebuild in months. The apps least likely to get pushed to are the apps most likely to be carrying a stale base layer — which is backwards from a security standpoint, since a quiet app is also the one nobody's watching closely enough to notice it's overdue.

What kpack actually watches

kpack's answer is to make the base image a first-class, reconciled resource instead of a value that gets read once and forgotten. It does this with a small chain of CRDs:

  • ClusterStore — the set of buildpack packages available to build with.
  • ClusterStack — the build-time and run-time base images (the OS layer buildpacks build against and apps run on top of).
  • Builder / ClusterBuilder — combines a stack, a store, and an order of buildpacks into a concrete "here's what a build looks like" resource.
  • Image — the per-app resource that says "build this source, with this builder, and keep it built."

The mechanism that makes automatic rebuilds possible lives in how Builder and ClusterBuilder track their own freshness: each carries observedStackGeneration and observedStoreGeneration fields, which record the generation of the ClusterStack and ClusterStore it last reconciled against. When someone publishes a new base image — a patched run image, a patched build image, an updated buildpack in the store — the ClusterStack or ClusterStore's generation advances.

kpack's controller notices the drift between the Builder's observed generation and the current one, reconciles the Builder against the new base, and then the Image controller — seeing its Builder has moved — kicks off a new Build for every app that references it. No source diff required. The base layer changing is the trigger.

That reconciliation loop is what turns "a CVE landed in the base image" from a fleet-wide announcement — someone has to notice, tell every team, and every team has to remember to push a no-op commit — into something the cluster does to itself.

Rebuild vs. rebase — the distinction that makes this cheap enough to run constantly

Watching the base layer only works operationally if reacting to it is cheap, and this is where kpack makes a distinction that's easy to gloss over but does most of the practical work: not every base image change deserves a full rebuild.

If only the run image changed — a pure OS package patch, no change to the build image or the buildpacks that ran — kpack can rebase instead of rebuild. A rebase swaps the base layer in the image manifest directly; it doesn't re-execute the buildpacks lifecycle (detect, build, export) at all, because none of the layers the buildpacks actually produced have changed. That's a manifest operation measured in seconds with no compute spent recompiling or reinstalling dependencies. A full rebuild, by contrast, is needed when the build image or the buildpack set itself changed — something that could plausibly change what gets produced — and that means paying for the full lifecycle again, typically minutes rather than seconds.

This is the piece that makes "watch the base layer and react automatically" viable at Paketo's actual patch cadence — 48 hours for High/Critical CVEs, a full rebuild of every stack weekly regardless — instead of theoretically nice but too expensive to run for real. A controller that had to do a full rebuild for every OS patch across an entire fleet would either fall behind or burn a lot of compute keeping up. A controller that can rebase for the common case and only pay full rebuild cost when the buildpacks themselves changed can plausibly keep an entire fleet current on a weekly-to-48-hour cycle without it becoming its own infrastructure problem.

What this would take on top of a Cluster API fleet's buildpacks pipeline

kpack itself is a single-cluster tool — it reconciles Image resources against Builders in the cluster it runs in. Porting the pattern, not the binary, onto a Cluster API fleet running its own buildpacks pipeline (the situation bex is actually in) means answering four concrete questions kpack already answers for a single cluster:

  1. A per-app resource that remembers what it was last built against. Today, a build only knows the source commit it built from. Closing this gap means also recording the base-image digest (or ClusterStack-equivalent generation) that build used — the thing a watcher needs in order to know a given app is now stale relative to the base image, not just relative to its own source.
  2. Something watching the base image source for new digests. Whether that's polling the upstream stack registry on an interval or reacting to a webhook from wherever the fleet's base images get published, this is the piece that currently doesn't exist at all in a push-only pipeline — nothing today asks "has the base image changed" independent of "did someone push code."
  3. A rebase-vs-rebuild decision, not just a rebuild trigger. Wiring in only "rebuild everything when the base changes" reproduces the fleet-wide manual-campaign cost the reconciliation loop was supposed to remove — it just automates the expensive path instead of replacing it. The cheap path (manifest-level rebase for pure base-image patches) is what makes running this constantly, across every tenant app, affordable.
  4. A shared entry point with the existing push-triggered build. This has to feed the same Build resource and the same rollout path a git push already triggers — it's an additional trigger on the existing pipeline, not a second deployment system living next to it. A base-image-triggered rebuild that rolls out through a different path than a push-triggered one is two things to operate and debug instead of one.

None of these four are exotic — kpack has already worked out the shape of each one for a single cluster. The work in a Cluster-API-fleet context is translating "watch a ClusterStack's generation" into "watch a base image across however many workload clusters a tenant's apps are spread across," and making sure the rebase fast path survives that translation, since it's the part doing the cost control.

The base layer is part of your surface area, whether or not anything is watching it

Buildpacks itself just crossed a real maturity marker: the CNCF moved the project from Incubating to Graduated status on July 17, 2026, nearly six years after it entered incubation. That's not a detail about kpack specifically, but it's a signal about the ecosystem kpack sits in — the tooling around "what's actually in your base image, and how current is it" is consolidating rather than staying a pile of bespoke scripts per platform.

The underlying point holds regardless of which buildpacks implementation a platform uses: a base image is part of an app's attack surface for as long as that app runs, not just at the moment it was built. A pipeline that only reconsiders the base layer when a developer happens to push is treating a continuous liability as a one-time decision. The apps that go quietest are the ones most exposed by that gap, precisely because nothing else is prompting anyone to look.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex