Argo CD 3.3 PreDelete Hooks: Stop Tenant Teardowns From Leaving Orphaned Wrecks

Everyone tests the deploy path. Almost nobody tests the delete path — until the day removing a tenant's application tree from git leaves its namespace, PVCs, and load-balancer DNS records running orphaned in the cluster for two days, unmonitored and still billing. That is a real incident from a Talos homelab operator: 28 Applications without the cascade-delete finalizer, one git rm, and an openclaw stack that kept running long after Argo CD forgot it existed.

A second operator had the opposite failure. Their ApplicationSet-managed tree did have finalizers — and when the source path vanished from git, the ApplicationSet controller called delete on the child Application every 15 seconds for 45 minutes while the finalizer could never finish, because the manifests it needed were gone. A ConfigMap survived, orphaned, in the middle of a deletion loop that looked healthy from the outside.

Deletion in the app-of-apps pattern has always been the sharp edge of Argo CD, and Argo CD 3.3 finally gave it a guard: PreDelete hooks. A PreDelete hook is a Kubernetes Job (or Pod) that must run to completion before Argo CD removes the rest of an Application's resources — the place to drain traffic, export data, or delete operator-managed CRs in the right order. This post names the three concrete ways app-of-apps teardown goes wrong, shows what PreDelete hooks change, lists the four gotchas that will bite you anyway, and answers whether Flux's simpler model is the better fit for a Cluster-API fleet.

The three ways app-of-apps deletion goes wrong​

The app-of-apps pattern — one root Application whose manifests are nothing but child Applications — is how every serious Argo CD fleet scales. It is also a deletion bomb with three distinct fuses. Here they are, with what each one concretely leaves behind:

Failure 1 is the most common and the most embarrassing. By default, deleting an Argo CD Application deletes only the Application object — the workloads stay. Cascade deletion is opt-in per Application via the finalizer, which means every fleet has a long tail of Applications somebody created by hand, through a Helm chart, or via an ApplicationSet template that forgot the finalizer. The fix is a one-line audit (kubectl get applications -A minus the ones carrying the finalizer), but nobody runs it until the orphan bill arrives.

Failure 2 is the one PreDelete hooks were built for. A cascade delete removes resources roughly in parallel, with no notion of "the database backup must finish before the StatefulSet goes away" or "the Istio VirtualService must drain before the Gateway disappears." Stateful leftovers are the classic casualty: Kubernetes never auto-deletes a StatefulSet's volume-claim-template PVCs, and Argo CD's prune does not track them either, so tenant data volumes routinely outlive the tenant. External state is worse — DNS records, bucket contents, cloud load balancers created by operators — because no in-cluster cascade can reach them at all.

Failure 3 is the subtlest. An ApplicationSet that generates Applications from git or cluster generators will prune a generated Application when its source disappears — and that prune path does not execute PreDelete hooks, which fire only on explicit Application deletion. Design your teardown cleanup as a hook and then delete the tenant by removing their directory from git, and your cleanup never runs. The runbook has to know which of the two deletion paths it is using, because Argo CD treats them differently.

What PreDelete hooks actually change​

Before 3.3, the hook lifecycle ended at sync time: PreSync, Sync, PostSync, plus SyncFail, and PostDelete (which runs after resources are gone — useful for notifications, useless for ordered cleanup). There was no way to say "run this Job first, and do not touch anything else until it succeeds." Operators filled the gap with external scripts, CI jobs that ran before the git rm merged, or manual runbooks — all of which drift out of sync with the cluster state they are supposed to protect.

A PreDelete hook closes that gap declaratively, inside the Application itself:

apiVersion: batch/v1
kind: Job
metadata:
  name: tenant-teardown
  annotations:
    argocd.argoproj.io/hook: PreDelete
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: teardown
          image: registry.example.com/platform/teardown:1.4.0
          args: ["--tenant", "acme", "--export-backup", "--drain-mesh"]

The semantics are deliberately strict, and that strictness is the feature:

• It blocks. Argo CD will not begin deleting the Application's remaining resources until every PreDelete hook reports success. A failed hook blocks the entire Application deletion — which is exactly what you want when the hook is "export the tenant's database," and exactly what will page you when the hook image has a typo.
• It runs with resources present. Unlike PostDelete, the hook executes while the Application's workloads still exist, so it can query them, drain them, snapshot them, and delete operator CRs in dependency order.
• It lives in git. The hook is a manifest in the same Application, reviewed and versioned like everything else — not a wiki page describing a script on somebody's laptop.

The version floor is Argo CD 3.3 or later (early 2026; OpenShift GitOps 1.20+ carries it downstream). The 3.3 release framed it as part of a broader deletion-safety push, and 2026 field reports consistently reach for the same three use cases: data export before StatefulSet teardown, traffic draining in service meshes, and notifying external systems — backup vaults, billing, status pages — of a service's retirement. Deletion becomes a governed lifecycle phase instead of the absence of sync.

The four gotchas (read before you ship a hook)​

PreDelete hooks are a safety net with holes you should know about in advance. All four are documented behaviors or open upstream issues, not FUD:

1. Hooks fire only on explicit Application deletion — not on ApplicationSet auto-prune. This is failure mode 3 from the table, restated as a rule: if your tenant offboarding removes a directory from git and lets the ApplicationSet generator prune the child Application, your PreDelete hook never executes. Either offboard tenants by explicitly deleting the Application object (argocd app delete or kubectl delete application), or put the prune-path cleanup in the generator's own lifecycle, not in a hook. Test the path you actually use, not the path in the demo.

2. A failed hook blocks deletion indefinitely — so bound your Jobs. "Blocks until it succeeds" includes "blocks forever while CrashLoopBackOff-ing." Every PreDelete Job needs a backoffLimit and an activeDeadlineSeconds, and your alerting needs a rule for "Application deletion stuck behind a failing hook," or a typo'd image tag turns a tenant teardown into a permanently wedged namespace. The deadlock incident in the intro — 45 minutes of delete calls against an uncompletable finalizer — is the shape of what an unbounded hook failure looks like from the outside.

3. Hooks are skipped during selective syncs. A sync --resource or --label targeted sync skips all hooks, including PreDelete. That is by design (you asked for one resource, not the lifecycle), but it means a partially-synced Application can sit in a state its authors assumed hooks would prevent. Do not use selective sync as a teardown shortcut.

4. The hook object itself races the cascade it gates. Upstream issue argoproj/argo-cd#29100 documents the sharpest edge: under cascade deletion, the controller can create the PreDelete hook resource, declare the pre-delete phase complete, remove the finalizer, and start cascade deletion — which garbage-collects the still-starting hook before it ever executes. Deletion then proceeds as if the hook ran. The practical mitigations are hook-delete-policy: BeforeHookCreation hygiene, verifying hook completion in your teardown runbook rather than assuming it, and tracking the upstream fix before you bet tenant data exports on the mechanism. A safety net you cannot observe is a hope, not a control — watch argocd app get hook status the way you watch sync status.

None of these is a reason to avoid PreDelete hooks. They are reasons to treat the hook as one component of a teardown runbook rather than the whole runbook — which is the next section's job.

The Flux fit check: is "no hooks" actually simpler?​

Flux deliberately has no equivalent of PreDelete hooks, and the comparison is instructive because the two projects make opposite bets about where teardown intelligence should live. Flux's prune: true on a Kustomization removes resources deleted from git during normal reconciliation — no finalizer audit needed, no opt-in per object. Suspending reconciliation freezes a tree in place. There is no central API server, no Redis, no UI service; each cluster's controllers converge that cluster from its sources, and multi-tenancy rides native Kubernetes RBAC.

For teardown specifically, that means: Flux cannot run ordered pre-delete cleanup at all — there is no hook point — but it also cannot strand you on a missing finalizer, because pruning is the default behavior of the reconciler rather than an annotation you might have forgotten. The failure modes compress to one: whatever prune removes, it removes in the reconciler's order, with no place to inject "export first."

So the decision for a Cluster-API fleet comes down to what your teardown actually contains:

• If tenant teardown is "delete the namespace tree, nothing external" — stateless apps, ephemeral review environments — Flux's prune model has fewer moving parts and no finalizer audit. The absence of hooks is a non-issue because there is nothing order-sensitive to run.
• If teardown touches state or external systems — tenant Postgres volumes, mesh draining, DNS/LB cleanup, backup exports — Argo CD with PreDelete hooks is the only one of the two that models the problem. But you must operate it as designed: finalizers on every Application, explicit-delete offboarding path, bounded hook Jobs, and hook-completion monitoring.

One hard rule regardless of choice, unchanged from every honest GitOps comparison: never run both reconcilers against the same namespaces. Two controllers managing the same objects do not add redundancy; they fight — each reverting the other's writes — until someone pages. One reconciler per cluster, chosen deliberately.

For a self-hosted PaaS on Cluster API specifically, there is a second-order effect worth naming. A platform that already centralizes fleet state in its own control-plane API — machine lifecycle, tenant records, billing — does not need GitOps to be the system of record for "tenant acme is gone." It needs GitOps to converge the cluster to that decision reliably, including the ordered cleanup. That pushes toward Argo CD with hooks for the tenant-facing trees (where state lives) even if the platform's own machine-management layer reconciles elsewhere. Use Flux where teardown is trivially prunable; use Argo CD with PreDelete gates where teardown has a blast radius.

A tenant-teardown runbook you can steal​

Putting it together — the checklist for retiring a tenant's full application tree on a self-hosted PaaS, assuming Argo CD manages the tenant trees:

1. Audit finalizers first. Confirm every Application in the tenant tree carries resources-finalizer.argocd.argoproj.io (or the :background variant). Fix the stragglers before the teardown, not during.
2. Ship the PreDelete hook with the tenant's stack, not at teardown time. Data export, mesh drain, external-system notification — reviewed in git alongside the workloads they protect, with backoffLimit and activeDeadlineSeconds set.
3. Offboard via explicit Application deletion, root first, so hooks fire. Do not rely on removing git paths and hoping the ApplicationSet prune path runs your hooks — it will not.
4. Watch hook status to completion (argocd app get, or your platform's own status surface) before considering the teardown done. Verify the backup artifact exists; verify the DNS record is gone.
5. Sweep for the untracked. StatefulSet PVCs, operator-provisioned external resources, and anything with prune: false — list them per tenant template and check them off explicitly. The hook handles the ordered; the sweep handles the invisible.

Argo CD spent years as a deployment tool with a deletion footnote. PreDelete hooks promote teardown to a first-class lifecycle phase — with real rough edges (the auto-prune gap, the unbounded-block failure mode, the upstream cascade race) that a runbook, not a version bump, has to cover. For a platform whose tenants trust it with state, that runbook is now part of the product: anyone can git push an app into existence, but safe, complete, provable removal is what separates a PaaS from a script that applies YAML.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.