Data Center / Cloud
Start and Restore GPU Workloads in Seconds with NVIDIA Snapshot
Nov 10, 2026
By Ron Kahn, Dan Feigin, Oz Bar-Shalom, Ekin Karabulut and Schwinn Saereesitthipitak
Like
Discuss (0)
L
T
F
R
E
AI-Generated Summary
NVIDIA Snapshot is an open-source checkpoint and restore system for GPU workloads on Kubernetes that captures fully initialized CPU and GPU memory state.Restoring from a Snapshot artifact brings inference replicas online in 3.6 to 42.7 seconds across tested configurations, delivering 8x to 36x speedups over cold starts that take minutes.The system supports multi-GPU workloads using tensor parallelism, data and expert parallelism, and decode context parallelism for large mixture-of-experts models with checkpoints up to 1 TB across eight GPUs.Checkpoint creation takes 15 seconds to 5.5 minutes depending on model size, and the combined checkpoint-plus-restore time is shorter than a single cold start in every tested configuration.Benchmark results on NVIDIA B200 GPUs with VAST NFS storage show that restoring GPU state dominates large restores at 18 to 29 seconds, while CRIU-based CPU process restore takes 4.2 to 10.8 seconds.Next StepsExplore the NVIDIA Snapshot repository for installation instructions and usage guides.Follow the NVIDIA Dynamo user guide to restore workers with Snapshot.Join the NVIDIA Dynamo Slack workspace to discuss integrations or provide feedback in the #snapshot-dev channel.
Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more
NVIDIA Snapshot, a new open-source component of the NVIDIA DSX AI Factory Platform, makes initialized GPU workloads reusable across inference replicas—speeding startup and recovery.
When inference demand spikes or a replica fails, available GPUs don’t translate immediately into serving capacity. Initializing an LLM workload—a process known as a cold start—can take minutes, delaying scaling and recovery.
Snapshot is a checkpoint and restore system for GPU workloads on Kubernetes. It captures a fully initialized workload—including CPU and GPU memory—and restores that state on compatible infrastructure, bringing inference capacity online in seconds instead of minutes. Figure 1 shows the startup sequence. Snapshot skips repeated workload initialization; infrastructure provisioning and container image loading remain outside the measured restore path.
Figure 1. Cold-start sequence: provision the machine, load the container image, start the inference engine, load model weights into GPU memory, compile with torch.compile, and capture NVIDIA CUDA graphs
This post explains how Snapshot checkpoints and restores GPU workloads, how platform teams can integrate it, and how restore times compare with cold starts across three inference engines.
Snapshot for AI factory operations
Originally developed for NVIDIA Dynamo inference workers, Snapshot provides checkpoint and restore capabilities that other inference engines, schedulers, controllers, and platforms can integrate.
Snapshot provides two operations for GPU Pods:
Checkpoint: Once a workload is fully initialized, capture its CPU and GPU memory as a portable artifact.
Restore: Start a new Pod from that artifact on any node with the same GPU model and driver. The process continues from where it was checkpointed, so weight loading, compilation, and graph capture are skipped.
Both operations use Kubernetes custom resources. To checkpoint a Pod that is already running, create a PodSnapshot that names it. To start a Pod from a template and checkpoint it as soon as it is ready, create a SnapshotJob instead. To restore, add the annotation nvidia.com/restore-from to a new Pod, naming the PodSnapshot to restore from. The workload only needs a small handshake to pause before the checkpoint and resume after the restore. The repository has ready-made examples for vLLM, SGLang, and NVIDIA TensorRT LLM.
Snapshot can checkpoint and restore compatible workloads that span multiple GPUs on a single node as a single checkpoint artifact across ranks. This supports fast startup for large mixture-of-experts (MoE) models using parallelism strategies such as tensor parallelism (TP), data and expert parallelism (DEP), and decode context parallelism (DCP). Workloads must meet Snapshot’s checkpoint/restore and runtime compatibility requirements. See the example configurations.
Benchmarks
Design partners have also evaluated Snapshot on their own workloads:
Crusoe, an upstream contributor to Snapshot, reduced GLM-5.3 Pod readiness time from roughly 8 minutes to 31 seconds on NVIDIA HGX B300.
Saturn Cloud, a token factory platform, benchmarked NVIDIA Snapshot for scale-from-zero inference, cutting vLLM startup from 2–8.5 minutes to 6–34 seconds on NVIDIA HGX B300.
The following benchmarks show how Snapshot accelerates inference startup across vLLM, SGLang, and NVIDIA TensorRT LLM (TRT-LLM)—with models from 600 million parameters to mixture-of-experts (MoE) models with 1 TB checkpoints on 8 GPUs. Across all tested configurations, Snapshot brings a replica back in under 45 seconds, 8x to 36x faster than a cold start.
Methodology
Each configuration was tested using two strategies on the same hardware and software stack:
Cold start: The workload starts from scratch. The clock runs from process launch until engine initialization returns, before inference. The measurement is divided into four phases (Table 2):
Startup: Starting the engine and worker processes, importing Python libraries, and setting up communication between GPUs. Some just-in-time (JIT) kernel compilation also happens here.
Model load: Reading the model weights and constructing the model’s tensors, including any engine-specific conversion or repacking done while loading.
Initialization: Preparing the loaded model to serve: memory profiling, allocating the key-value (KV) cache and runtime buffers, and other kernel and runtime preparation.
Compilation, autotuning, and graph capture: Compiling kernels (for example, with torch.compile), autotuning them for the GPU, and capturing CUDA graphs.
Snapshot: The workload is restored from its checkpoint. The clock runs from the moment Snapshot is asked to restore until the workload is serving again. The measurement is divided into four stages (Table 3):
Agent setup: The Snapshot agent on the node locates the checkpoint artifact and prepares the target Pod and GPUs.
Checkpoint/Restore in Userspace (CRIU) restore: CRIU, the standard Linux tool for checkpointing processes, reads the process image from storage into host memory. Speed is set mostly by the storage backend.
CUDA restore: Copying GPU state back into GPU memory.
Wake/remap: Adapting the process to the new node and waking the engine.
Both measurements exclude pulling the container image and setting up the container runtime. The KV cache release is triggered before every checkpoint, so it is not part of the artifact.
Table 1 also reports checkpoint time and size: how long it takes to capture a fully initialized replica and how much storage the stored artifact requires. This is a one-time cost per model and configuration.
Setup
GPU: NVIDIA B200
NVIDIA driver: 615.71.09
Storage backend: VAST (NFS)
Inference engines:
vLLM 0.31.0
SGLang 0.5.21
TensorRT LLM 1.3.0
Results
The benchmarks show four main findings.
Restore times stayed under 43 seconds: Across all tested configurations, Snapshot restored replicas in 3.6 to 42.7 seconds—8x to 36x faster than cold start. The largest MoE models, with checkpoints of about 600 GB to 1 TB across eight GPUs, restored in under 43 seconds compared with cold-start times of 6 to 19 minutes.
Restoring skips more than weight loading: Cold-start time includes loading weights, compiling and autotuning kernels, and capturing CUDA graphs. Snapshot restores the initialized workload, skipping these repeated steps.
Checkpointing pays off with the first restored replica: In every tested configuration, checkpoint time plus restore time was shorter than a single cold start. Checkpoint creation took 15 seconds to 5.5 minutes, depending on the model and configuration.
Moving GPU state dominates large restores: For the eight-GPU MoE configurations, restoring GPU state took 18 to 29 seconds. Restoring CPU process state with CRIU took 4.2 to 10.8 seconds.
Figures 2 to 4 compare cold-start and Snapshot restore times for vLLM, SGLang, and TensorRT LLM across the tested models and parallelism configurations. Table 1 provides the data for every configuration, including checkpoint time and size. All times are in seconds. In the figures and tables, TEP8 denotes tensor and expert parallelism across eight GPUs.
Figure 2. vLLM: Cold start compared with Snapshot for every model and parallelism tested. Snapshot brings each replica up in 4.7 to 39.6 seconds
Figure 3. SGLang: Cold start compared with Snapshot for every model and parallelism tested. Snapshot brings each replica up in 3.6 to 42.7 seconds
Figure 4. TensorRT LLM (TRT-LLM): Cold start compared with Snapshot for every model and parallelism tested. Snapshot brings each replica up in 7.79 to 33.55 seconds
ModelEngineParallelismWeight size (GB)Checkpoint size (GB)Checkpoint time (s)Cold start (s)Snapshot agent restore (s)SpeedupQwen3-0.6BvLLMTP11.508.1315.30104.434.6622.40xQwen3-0.6BSGLangTP11.509.4220.7187.083.5924.26xQwen3-0.6BTRT-LLMTP11.508.0420.07121.268.1514.88xQwen3-14BvLLMTP129.5437.7119.98122.765.7221.45xQwen3-14BvLLMTP229.5448.2938.37139.636.1922.55xQwen3-14BSGLangTP129.5440.1926.17106.265.6018.96xQwen3-14BSGLangTP229.5451.9333.88124.727.4516.75xQwen3-14BTRT-LLMTP129.5436.3823.72127.0212.5010.16xQwen3-14BTRT-LLMTP229.5447.9742.90158.6615.999.92xgpt-oss-120bvLLMTP165.2581.5644.37172.8010.0917.12xgpt-oss-120bvLLMTP465.25119.69133.01155.6611.6213.40xgpt-oss-120bSGLangTP165.25172.3255.63268.9615.9416.87xgpt-oss-120bSGLangTP465.25693.89174.20240.1729.648.10xgpt-oss-120bTRT-LLMTP165.2577.8729.98200.557.7925.75xgpt-oss-120bTRT-LLMTP465.25111.3485.29223.9618.5812.05xGLM-5.3-NVFP4vLLMTEP8464.82604.00282.80393.6528.6413.74xGLM-5.3-NVFP4SGLangTEP8464.82669.49327.20426.7035.3412.07xGLM-5.3-NVFP4TRT-LLMTEP8464.82628.30231.33363.9033.5510.85xDeepSeek-V4.1-FlashvLLMTEP8510.30647.18274.751110.6530.9635.87xDeepSeek-V4.1-FlashSGLangTEP8510.30743.35286.77953.0936.5426.08xDeepSeek-V4-Pro-NVFP4vLLMTEP8913.09996.53193.84908.4839.5622.96xDeepSeek-V4-Pro-NVFP4SGLangTEP8913.091039.09253.72621.5542.6914.56xTable 1. Cold start compared with a Snapshot restore on NVIDIA B200 with VAST (NFS) storage, on 1 to 8 GPUs. Checkpoint time is a one-time cost per configuration. Speedup is cold start time divided by Snapshot restore time
Table 2 breaks down the cold-start time, showing which initialization phases a restore skips.
ModelEngineParallelismStartup (s)Model load (s)Initialization (s)Compilation, autotuning & graph capture (s)Total (s)Qwen3-0.6BvLLMTP131.3717.2714.5841.21104.43Qwen3-0.6BSGLangTP133.7411.678.4133.2687.08Qwen3-0.6BTRT-LLMTP125.793.7940.6451.04121.26Qwen3-14BvLLMTP131.1626.1515.3050.15122.76Qwen3-14BvLLMTP242.1818.9615.2963.20139.63Qwen3-14BSGLangTP133.4520.798.2843.74106.26Qwen3-14BSGLangTP247.6218.938.2949.88124.72Qwen3-14BTRT-LLMTP125.3511.5439.3650.77127.02Qwen3-14BTRT-LLMTP227.428.8870.6251.74158.66gpt-oss-120bvLLMTP135.1538.6910.4988.47172.80gpt-oss-120bvLLMTP449.9124.7112.1368.91155.66gpt-oss-120bSGLangTP142.5434.440.36191.62268.96gpt-oss-120bSGLangTP457.9749.061.33131.81240.17gpt-oss-120bTRT-LLMTP126.2147.5042.8583.99200.55gpt-oss-120bTRT-LLMTP434.2842.7222.34124.62223.96GLM-5.3-NVFP4vLLMTEP865.4765.8848.88213.42393.65GLM-5.3-NVFP4SGLangTEP860.7773.982.66289.29426.70GLM-5.3-NVFP4TRT-LLMTEP831.5890.39120.43121.50363.90DeepSeek-V4.1-FlashvLLMTEP863.96184.9077.25784.541110.65DeepSeek-V4.1-FlashSGLangTEP856.1874.2138.50784.20953.09DeepSeek-V4-Pro-NVFP4vLLMTEP864.4295.1464.87684.05908.48DeepSeek-V4-Pro-NVFP4SGLangTEP858.8087.6842.55432.52621.55Table 2. Where the cold start time goes, by phase, in seconds. Phase timings may differ from the total because of rounding or intervals not captured by logging and instrumentation
Table 3 breaks down restore time into agent setup, CRIU restore, CUDA restore, and wake/remap.
ModelEngineParallelismSnapshot: agent setupSnapshot: CRIU restoreSnapshot: CUDA restoreSnapshot: wake/remapTotal (s)Qwen3-0.6BvLLMTP12.301.360.780.224.66Qwen3-0.6BSGLangTP10.241.331.020.993.59Qwen3-0.6BTRT-LLMTP15.721.430.840.168.15Qwen3-14BvLLMTP10.131.583.160.855.72Qwen3-14BvLLMTP20.572.033.430.176.19Qwen3-14BSGLangTP10.241.353.670.345.60Qwen3-14BSGLangTP20.922.173.490.877.45Qwen3-14BTRT-LLMTP16.802.702.820.1812.50Qwen3-14BTRT-LLMTP29.653.522.410.4115.99gpt-oss-120bvLLMTP10.482.606.710.3010.09gpt-oss-120bvLLMTP41.024.405.570.6211.62gpt-oss-120bSGLangTP10.261.6913.460.5415.94gpt-oss-120bSGLangTP42.694.3421.690.9329.64gpt-oss-120bTRT-LLMTP10.281.515.740.267.79gpt-oss-120bTRT-LLMTP49.993.304.340.9518.58GLM-5.3-NVFP4vLLMTEP83.794.1820.610.0628.64GLM-5.3-NVFP4SGLangTEP83.2710.8019.961.3135.34GLM-5.3-NVFP4TRT-LLMTEP89.815.3218.170.2533.55DeepSeek-V4.1-FlashvLLMTEP84.845.7119.371.0430.96DeepSeek-V4.1-FlashSGLangTEP84.349.7721.401.0336.54DeepSeek-V4-Pro-NVFP4vLLMTEP85.444.7328.980.4239.56DeepSeek-V4-Pro-NVFP4SGLangTEP83.1310.1328.600.8342.69Table 3. Where the Snapshot restore time goes, by stage, in seconds. Together, the four stages make up the Snapshot column in Table 1
Figure 5 shows the restore-time breakdown for vLLM.
Figure 5. Snapshot restore time by stage on vLLM, shown as an example. The number above each column is the total restore time in seconds
Benchmarking tools and reproduction instructions are available in the repository.
Get started
To try Snapshot with a GPU workload on Kubernetes, start with the installation instructions and usage guides in the NVIDIA Snapshot repository.
Try it out with NVIDIA Dynamo to restore workers by following the user guide.
To discuss an integration or contribute feedback, open an issue or join the NVIDIA Dynamo Slack workspace and visit the #snapshot-dev channel.
Visit the NVIDIA booth #170 at KubeCon for Kubernetes deep dives and live demos.
Discuss (0)
Like
Tags
AI Platforms/Deployment | Data Center / Cloud | MLOps | General | DSX | Intermediate Technical | Benchmark | Cloud Services | featured | KubeCon | Kubernetes | Open Source
About the Authors
About Ron Kahn
Ron Kahn is a senior software engineer at NVIDIA and a maintainer of Grove and Snapshot. He works on Kubernetes infrastructure for AI inference, focusing on reducing GPU cold starts, speeding container recovery, and orchestrating multinode inference workloads.
View all posts by Ron Kahn
About Dan Feigin
Dan Feigin is a senior software engineer at NVIDIA with an emphasis on container state recovery and GPU checkpoint/restore mechanisms. He focuses on developing solutions utilizing Snapshot restore, CRIU, and CUDA to minimize startup times for large-scale inference services and fractional GPU workloads on Kubernetes.
View all posts by Dan Feigin
About Oz Bar-Shalom
Oz Bar-Shalom is a product manager with a decade of experience in cloud infrastructure. At NVIDIA, he works on the Run:ai product, helping organizations optimize their AI infrastructure. He focuses on challenges in inference, multi-tenancy, and infrastructure compatibility, driving solutions that make complex AI systems accessible and effective for organizations running workloads at scale. Previously, he was a product manager at Run:ai and held engineering leadership roles at Oosto, where he built AI-based products.
View all posts by Oz Bar-Shalom
About Ekin Karabulut
Ekin Karabulut is a data scientist and developer advocate previously at Run:ai, now at NVIDIA, exploring the efficient usage of large models in different production scenarios. Previously she worked on privacy implications of federated learning, focused on distributed training techniques and got fascinated by inefficiencies in GPU usage in research and industry settings. She established the AI Infrastructure Club and is based in Munich, Germany.
View all posts by Ekin Karabulut
About Schwinn Saereesitthipitak
Schwinn Saereesitthipitak is a Software Engineer at NVIDIA specializing in large-scale distributed inference systems. His work bridges research and engineering across the inference stack to enable elasticity and resiliency of production workloads at scale.
View all posts by Schwinn Saereesitthipitak
Comments
精选
暂无精选,请稍后再看。
