Cold start latency is a major bottleneck for AI inference workloads on Kubernetes. When demand spikes, new replicas must pull container images, load model weights into GPU memory, warm up CUDA kernels, compile or capture CUDA graphs, and register with service discovery before they can serve any request. During this several‑minute window, GPUs sit idle, wasting resources and risking SLA violations.
NVIDIA Dynamo Snapshot solves this by checkpointing a running inference worker and restoring it instantly, bypassing the full cold‑start sequence. The approach splits the worker’s state into device‑side (GPU) and host‑side (CPU) components. First, cuda‑checkpoint uses the CUDA driver to dump GPU contexts, streams, device memory, and virtual address mappings into CPU memory. Then CRIU (Checkpoint/Restore in Userspace) walks the Linux kernel’s bookkeeping and serializes the process tree—CPU memory, threads, file descriptors, and namespaces—to shared storage. On restore, the order is reversed: CRIU rebuilds the process tree from storage, and cuda‑checkpoint places the GPU state back onto the new GPUs.
Several practical optimizations shrink checkpoint size and speed up restore. The KV cache, allocated after weights and CUDA graphs, does not need to be saved because no requests have been served yet. By using cuMemCreate and cuMemMap to reserve the virtual address range and then cuMemUnmap plus cuMemRelease to free the physical pages, the artifact size drops from ~190 GiB to ~6 GiB for a Qwen3‑0.6B model on a B200 GPU.
CRIU restore time is further cut by two pending upstream enhancements: parallel memfd restore, which uses a thread pool to recover shared anonymous memory buffers concurrently, and Linux native AIO, which keeps up to 128 read requests in flight to saturate NVMe bandwidth. Together they deliver up to a 7.9× speedup over stock CRIU.
The GPU Memory Service (GMS) removes the remaining serial bottleneck by decoupling large model weights from the CRIU artifact via the CUDA Virtual Memory Management API. Process state and weight restoration then run in parallel over fast paths such as GPUDirect Storage or peer‑GPU RDMA/NVLink. In a proof‑of‑concept with eight striped local NVMe SSDs, gpt‑oss‑120b startup fell from ~119 seconds to under 5 seconds—a 21× reduction.
Deployment uses a privileged DaemonSet (snapshot‑agent) installed via Helm, a DynamoCheckpoint custom resource that identifies the model configuration, and a DynamoGraphDeployment that references the checkpoint for restore. Prerequisites are x86_64 GPU nodes with NVIDIA driver 580.xx+ (590.xx+ for multi‑GPU), ReadWriteMany storage, and the vLLM backend in limited preview.
Key takeaways: Dynamo Snapshot eliminates cold‑start latency, slashes checkpoint size through KV cache unmapping, accelerates restore with parallel memfd and AIO, enables concurrent weight and process restore via GMS, and integrates cleanly with Kubernetes for elastic, production‑grade inference scaling.
#AI #Product #Kubernetes #ML #Inference #NVIDIA