Dora NodaEvery on-call engineer knows the worst kind of crash loop: the container dies, the kubelet restarts it, the evidence dies with it. You get logs — if the app logged anything useful before it fell over — and a restart count climbing toward triple digits. The heap, the open file descriptors, the exact goroutine or thread that was stuck: gone, every single time.
There is a better primitive, and it already ships in your kubelet. One authenticated POST to the kubelet's checkpoint endpoint freezes a running container into a tarball — memory pages, process tree, open file descriptors, filesystem diff — sitting at /var/lib/kubelet/checkpoints/ on the node. Pull that archive off the machine, pick it apart with checkpointctl and crit, and you are debugging the failure as it actually looked mid-crash instead of reconstructing it from log lines after the fact. In January 2026 the Kubernetes project gave this whole space a formal home by announcing the Checkpoint/Restore Working Group. This post is the practical version of that announcement: what the working group covers, the exact checkpoint-debug-restore loop for a crash-looping pod, and the four boundaries where the technology still says no.
The one call that freezes a container
The Kubelet Checkpoint API (KEP-2008) arrived as alpha in Kubernetes v1.25 and graduated to beta in v1.30, which means the ContainerCheckpoint feature gate has been on by default for several releases now. The kubelet does not do the freezing itself: it delegates through the CRI CheckpointContainer call to the runtime, which shells out to CRIU — Checkpoint/Restore In Userspace, the decade-old Linux tool that dumps a live process tree to disk.
The call itself is one line against the kubelet's HTTPS port:
curl -sk -X POST \
--cacert /etc/kubernetes/pki/ca.crt \
--cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
--key /etc/kubernetes/pki/apiserver-kubelet-client.key \
"https://${NODE_IP}:10250/checkpoint/${NAMESPACE}/${POD}/${CONTAINER}"Seconds later the node holds a file named like checkpoint-myapp_production-web-7d9f8c6b9-xl2v4_app-2026-09-09T14-03-11Z.tar. What is inside is a full core dump with ambition. The archive carries the CRIU image directory (memory pages, process metadata, file-descriptor tables), a rootfs-diff.tar with every filesystem change since the container started, the container runtime's config.dump and spec.dump, and CRIU's own working logs under criu.work/. That last directory matters more than it looks: when a checkpoint itself misbehaves, the dump log is where you find out why.
Two properties of this design decide everything else in the post. First, checkpoint cost scales roughly linearly with the container's resident memory — a 200 MB web app freezes in seconds, a 40 GB inference server does not. Second, this is a node-local file, not a cluster object. Nothing replicates it, nothing garbage-collects it on a schedule you chose. Your runbook has to move it somewhere safe before the node is recycled, which is exactly the kind of unglamorous operational detail Section 5 exists for.
What the January 2026 working group actually announced
On January 21, 2026, Radostin Stoyanov, Viktória Spišaková, Adrian Reber, and Peter Hunt announced the Checkpoint/Restore Working Group — a standing forum for the Kubernetes community and the CRIU ecosystem to design this integration together, rather than as drive-by KEPs. The announcement names six use-case families, and they are worth reading as a map of where this technology is going, not just where it is:
- Shrinking the footprint of interactive workloads such as Jupyter notebooks and AI chatbots by checkpointing idle sessions to disk.
- Fast-starting applications with long initialization times — Java services and LLM inference servers — by restoring a pre-warmed checkpoint instead of replaying startup.
- Fault tolerance for long-running work like distributed model training via periodic checkpoints.
- Interruption-aware scheduling: preempt a low-priority pod without destroying its runtime state.
- Live pod migration across nodes for load balancing and maintenance.
- Forensic checkpointing: freezing a compromised or crashed container for post-mortem analysis.
The debugging story in this post is that last bullet. But notice what the list implies: the same mechanism that snapshots your crash loop today is the substrate for fast-start inference and preemptible batch jobs tomorrow. Learning the tooling now compounds.
The CRIU side of the table brings four projects: CRIU itself, checkpointctl for analyzing checkpoint archives, criu-coordinator for coordinated checkpointing of distributed applications, and the checkpoint-restore-operator for managing checkpoint schedules inside a cluster. The group meets every second Thursday at 17:00 UTC, with discussion on the #wg-checkpoint-restore Slack channel — relevant because several of the limitations in the next sections are open design questions there, not settled law.
The end-to-end loop: from crash loop to local reproduction
Here is the workflow, concretely, for a garden-variety crash-looping web app — say a tenant service OOM-flapping or wedged on a poison message.
Prerequisites. Three things must be true on the node, and all three are reasons this story favors self-hosted fleets: Kubernetes v1.30 or newer so the ContainerCheckpoint gate is on by default, CRIU installed on every node (criu check should pass), and a runtime that implements the CRI checkpoint call — CRI-O with enable_criu_support = true, or containerd 2.x. Note the containerd version carefully: 1.7 lacks the CheckpointContainer RPC entirely, so a fleet still on older node images gets a clean error, not a corrupt checkpoint.
Take the checkpoint while the failure is live. Time this against the crash loop: you want the container's restart backoff to leave the doomed process alive long enough to freeze. kubectl get pod -w shows you the window; the curl from Section 1 does the freezing. Copy the tarball off the node immediately — scp is fine, object storage is better — because a MachineHealthCheck remediation or a routine rolling upgrade will happily delete your only copy of the evidence.
Inspect without restoring. This is the step most teams skip, and it is the highest-value one. checkpointctl reads the archive directly:
checkpointctl show /tmp/checkpoint-myapp.tar
checkpointctl memparse --pid=1 /tmp/checkpoint-myapp.tar --output=/tmp/app-memory-pages.txtshow summarizes the process tree and resource usage captured at freeze time; memparse spills the actual memory pages for the PID you care about. For finer work, crit — the CRIU Image Tool bundled with CRIU — decodes individual image files: which file descriptors were open, what the socket table looked like, the exact process tree. At this point you are holding answers that logs almost never contain: the request body sitting in a buffer, the half-written response, the file descriptor leaked one connection at a time until the process hit its limit.
Restore locally and poke it. To bring the frozen container back, convert the archive into an OCI image and run it where a debugger is allowed:
checkpointctl build ./checkpoint-myapp.tar quay.io/example/myapp-checkpoint:latest
buildah push quay.io/example/myapp-checkpoint:latestA pod created from that image with the checkpoint annotations starts not from main() but from the frozen instant — the wedged state, reproducible on your laptop instead of on a production node at 3 a.m. That is the whole pitch: the crash loop becomes a deterministic local artifact instead of a story you piece together from timestamps.
The four boundaries where checkpointing still says no
An honest post prices the limitations before you hit them in production. There are four, and each has a concrete shape.
Open network connections are the classic wall. By default CRIU refuses to dump a process holding established TCP connections, because restoring a socket whose peer is on another host is genuinely hard. The escape hatch is TCP repair mode (tcp-established in /etc/criu/runc.conf), which freezes the socket with enough kernel state to re-establish it — on the same host, with a cooperative network namespace. Checkpoint a chatty microservice, restore it on a different node, and the restored sockets point at a conversation the other end has already forgotten. For debugging this is usually fine: you want the memory and the file descriptors, not a live connection. For migration it is the reason pod-level live migration remains a research project rather than a runbook.
GPU state needs a separate plugin and dominates the archive. Standard CRIU does not capture device memory. NVIDIA's path is the CRIU CUDA plugin plus cuda-checkpoint, with driver and version constraints (recent guides pin CRIU 4.1 or newer, NVIDIA driver 570 or newer, and the plugin on PATH). And the economics change completely: for a large model server, 90-plus percent of the checkpoint is GPU state, so the "checkpoint time scales with memory" rule from Section 1 now means gigabytes, dominated by disk I/O on the way out and device restore on the way back. Checkpointing a CPU web app for debugging is cheap; checkpointing a 40 GB inference server is a capacity decision.
Sandboxed runtimes are out of scope. CRIU needs deep host-kernel cooperation — process freezing, memory dumping, namespace surgery — that the gVisor syscall-interception layer and Kata's VM boundary deliberately do not provide. If your tenant isolation story is gVisor or Kata microVMs, container checkpointing is not available inside that boundary today. This is a real architectural fork for a multi-tenant platform: the runtimes with the strongest isolation are the ones this debugging superpower cannot reach.
Restore is container-level, not pod-level. The kubelet API freezes one container; restoring a whole multi-container pod with shared volumes and coordinated state is KEP-5823, still working through design. Sidecar-heavy pods — the service-mesh-injected, log-shipping, proxy-fronted shape that production pods actually have — checkpoint one container at a time, and reassembling the pod's joint state is your problem. For the canonical debugging target (one app container wedged while its logging sidecar hums along) this is exactly what you want. For stateful multi-container restores, wait for the KEP.
Why a self-hosted fleet gets this first
Here is the quiet advantage the TODO item was pointing at: every prerequisite in Section 3 is a node-level change. Install CRIU on the image. Run containerd 2.x. Keep the feature gate on. Reach the kubelet's port 10250 with a client certificate. On a managed control plane you often cannot do two or more of these — the provider owns the node image, the runtime version, and the kubelet's firewall rules. On a Cluster-API-managed fleet of machines you own, it is a machine-image change plus a runbook, full stop.
The fleet runbook practically writes itself. Bake CRIU into the node image and assert it in conformance (criu check in the image build, not as a prayer at 3 a.m.). Set a retention policy for /var/lib/kubelet/checkpoints — node-local disk fills fast when every crash loop leaves a memory-sized tarball behind. Ship a small script that wraps the curl, the scp to object storage, and a checkpointctl show summary into one on-call command, because nobody under incident pressure should be assembling kubelet client-cert flags from memory. And gate the whole thing on the same RBAC discipline as exec: whoever can checkpoint a container can read its memory, which means secrets, request bodies, and everything else the process was holding. Treat checkpoint archives as the most sensitive artifact your cluster produces — encrypt at rest, expire aggressively.
One honest caveat to close on: checkpointing does not fix your crash loop. It converts an unrepeatable production failure into a repeatable local artifact, which is maybe 80 percent of the debugging battle — the remaining 20 percent is still reading the code. But after years of debugging distributed systems by staring at the logs the dying process happened to emit, freezing the patient mid-seizure and examining it at leisure feels like a superpower. The working group exists to make it a boring, reliable one.
Want a fleet where node-level capabilities like this are one image change away? Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
