Skip to main content

Self-Hosted Kubernetes Installs Don't Stall on Kubernetes — They Stall on Your Registry, Network, and Approval Chain

9 min readDora NodaDora Noda
Share

On July 8, 2026, Fairwinds CTO Andy Suderman published a postmortem that names something most vendor case studies gloss over: why a self-hosted Kubernetes install that was scoped as a few days of work turns into weeks — sometimes months — of ad hoc debugging over three-hour calls with no clean way to predict when it'll end. His answer isn't "Kubernetes is hard." It's narrower and more useful than that: the thing that stalls these installs is almost never the application. It's the stack of dependencies sitting around it — a database nobody prepped, a registry the customer won't open write access to, a network path that looked fine in the architecture diagram and doesn't work in practice. That distinction matters beyond Fairwinds' own AI-platform customers, because it draws a line that every self-hosted PaaS vendor — bex included — needs to be honest about: "self-hosted" is not one claim. It's at least two, and only one of them is a problem bex has actually solved.

What Actually Stalls These Installs

Suderman's writeup is specific about the failure pattern, and specificity is what makes it useful instead of just another "enterprise software is hard" post. The blockers he names aren't exotic — they're the same handful of issues recurring across customers, each one small enough to sound trivial in isolation and large enough to burn days when it hits mid-install:

BlockerWhat actually happens
Database not readyThe install kicks off on schedule; the database it depends on isn't provisioned, isn't reachable, or isn't at the right version yet
Registry write access deniedThe customer's image registry policy doesn't permit the vendor to push images into it — discovered during the install, not before
Network path fails in practiceThe topology on the architecture diagram doesn't match what's actually routable; a "should just work" connection doesn't
Terraform treated as a guidelineCustomer teams use the vendor's IaC modules as a reference and then hand-build pieces in the cloud console, so the running environment silently drifts from what the modules describe
Security review surfaces something newA security team flags a concern mid-install that nobody on either side had surfaced during scoping
Prerequisites partially doneRequirements listed in the install doc are only partly complete by the time the session starts

None of these is a Kubernetes problem. Suderman's own framing is direct: "What actually stalls these deployments is the stack of dependencies around the application." Product engineers end up debugging issues that are part application, part infrastructure, and part customer environment — work that isn't really theirs to own, and that no amount of Kubernetes expertise on the vendor's side fixes, because the constraint isn't in the vendor's control. Multiply a handful of these per install across "some customers go live quickly, others take weeks, and in the most challenging cases installs drag on for months," and you get the pattern Suderman says "a lot of AI software companies are right now" living through — burning SE, PS, and support hours on work that looks like deployment but functions like discovery.

Two Different Things Both Called "Self-Hosted"

Here's why this postmortem is worth reading past Fairwinds' own customer base: it exposes that "self-hosted" quietly means two structurally different things, and the difference is exactly what determines whether Suderman's failure mode applies to you.

Self-hosted because you own the machines. This is bex's actual model today. bex provisions Cluster API-managed nodes on Hetzner hardware bex controls end-to-end — bex's own image registry, bex's own network topology, bex's own IaC applied by bex's own automation, no customer security team standing between a git push and a running service. There's no handoff, because there's no second party's infrastructure in the loop.

Self-hosted because a customer's compliance team demands it. This is what Fairwinds is actually describing: installing into infrastructure the vendor doesn't own and doesn't control. The customer's registry policy, the customer's network segmentation, the customer's security review process, the customer's internal approval chain across regions and teams — every one of Suderman's named blockers is a variable that belongs to someone else, and the vendor finds out where it breaks only by running into it.

Those two sentences sound similar. They are not the same install. Running the comparison blocker-by-blocker makes the gap concrete — and it's honest to note the sweep isn't clean:

Blockerbex's owned-Hetzner modelA hypothetical bex on-prem sale into customer infra
Database not readyDoesn't apply — bex doesn't manage tenant databases at all (a deliberate non-goal, not a gap)Applies in full — bex would depend on the customer's database being ready on the customer's timeline
Registry write access deniedDoesn't apply — it's bex's own registryApplies — bex would need write access to a registry it doesn't control
Network path fails in practiceMostly doesn't apply — the network between bex's own nodes is bex's to provision and verifyApplies — bex would inherit whatever network topology and segmentation the customer already has
IaC treated as a guidelineDoesn't apply — bex's own automation runs bex's own Cluster API manifests, with no customer hand-editing in betweenApplies — nothing stops a customer team from treating a provided IaC module as a starting point and drifting from it
Security review surprisesStill applies. A prospect's security team can flag a concern about bex itself — how it handles secrets, what it phones home to, its own supply-chain posture — with no customer infrastructure involved at allApplies, compounded — the customer's security review now also covers infrastructure bex doesn't operate
Prerequisites partially doneDoesn't apply — there's no customer-side prerequisite list to be incompleteApplies — exactly the failure mode Suderman describes

That one row that doesn't clear — security review — is the honest caveat: owning the hardware removes most of Fairwinds' blockers, not all of them. A prospect's compliance team can still ask hard questions about a vendor they don't operate, even when that vendor's infrastructure story is otherwise as clean as "we own every machine." The difference is that a security review of bex-as-vendor is a conversation bex can prepare for and answer directly; a security review of a customer's own registry, network, and approval chain is a conversation happening about infrastructure bex has never seen.

What a Cluster-API PaaS Should Learn Before It Ever Sells On-Prem

Today, none of this is bex's problem — the owned-Hetzner model structurally sidesteps five of Suderman's six blockers. It becomes bex's problem the moment bex (or any self-hosted PaaS built the same way) takes a sales call that ends with "install this into our own data center" or "deploy this air-gapped, on hardware we control." At that point, the postmortem stops being someone else's cautionary tale and becomes a checklist:

  1. Test registry write access before scheduling the install, not during it. A failed push on day one of a multi-week engagement is a policy conversation, not a debugging session — and it's a five-minute check to run ahead of time.
  2. Validate the actual network path, not the diagram. "Should be routable" and "is routable" are different claims; confirming the second before committing an install window catches the gap Suderman calls out — a path that "works on paper and not in practice" — before it burns a session.
  3. Don't trust that IaC modules were actually run. If a customer's environment is built partly by a provided Terraform module and partly by hand in a console, the running state has already drifted from what the module describes. A drift check against the module's expected state, run before install day, catches this before it looks like an application bug.
  4. Schedule the security review up front, not mid-install. Air-gapped and on-prem installs carry their own well-documented technical prerequisites — a registry the tool is actually allowed to write into, base images with the init system the bootstrap process expects, an internal time source since nodes can't reach public NTP (Kubernetes' certificate validation and etcd leader election are both sensitive to clock skew). None of those specifics matter if the security review that gates the whole install starts after the technical work is already underway.
  5. Gate the start date on a completed prerequisite checklist, not a best-effort one. Suderman's "prerequisites only partially done when the install begins" is the blocker most within a vendor's control to prevent — a go/no-go check the day before install, not a hope that everything on the list actually got done.

None of this is generic "run enterprise sales better" advice — it's a direct translation of Suderman's own list into the pre-flight gates a Cluster-API-based platform would need before it could credibly commit to a fixed install timeline on infrastructure it doesn't own. The broader context makes the stakes clear: industry data on enterprise on-prem software consistently puts sales cycles at six to twelve months against one to three for equivalent SaaS, and on-prem AI deployments built from scratch at three to six months versus weeks on a managed platform. Fairwinds' postmortem is the mechanism behind those numbers — not bad luck, but a predictable list of handoffs that nobody gated in advance.

The Boundary Worth Keeping Honest

The useful takeaway from Suderman's postmortem isn't "avoid on-prem installs forever." It's narrower: don't let a pitch built on "self-hosted because we own the machines" quietly borrow the confidence of a claim it hasn't earned — "self-hosted because your compliance team demands it." They sound like the same sentence. Fairwinds just spent a postmortem documenting how differently they actually play out. A platform that conflates the two in front of a prospect isn't lying, exactly — it's setting an installation-time expectation it doesn't yet have the process to meet, and finding out the gap exists on someone else's install-day call instead of on a checklist ahead of time.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, orchestrated by Cluster API on hardware you actually control. Star the repo on GitHub or deploy your first app today.

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex