CVE-2024-0132·Kernel vulnerability
NVIDIA Container Toolkit 1.16.1 or earlier contains a Time-of-check Time-of-Use (TOCTOU) vulnerability when used with default configuration where a specifically crafted container image may gain access to the host file system. This does not impact use cases where CDI is used. A successful exploit of this vulnerability may lead to code execution, denial of service, escalation of privileges, information disclosure, and data tampering.
- Severity
- critical
- Software
- Kernel
- Fixed in
- 24.6.2
- Published
- 2024-09-26
Affected versions
Until: 24.6.2
Fixed in: 24.6.2
How to fix this CVE
Upgrade NVIDIA Container Toolkit to version 1.16.2 or later and nvidia-container-runtime to 3.16.2 or later. If immediate upgrade is not possible, disable the allow-cuda-compat-libs-from-container CDI feature and ensure no untrusted container images are run with GPU access.
# Update the toolkit on all GPU nodes:
kubectl get nodes -l nvidia.com/gpu.present=true
# SSH to each node and update nvidia-container-toolkit
# Then restart containerd/docker and kubelet:
sudo systemctl restart containerd kubeletDefensia detects this vulnerability
What an exploitation attempt looks like
Sample log line indicative of exploitation attempts:
The attacker builds a malicious container image that exploits the TOCTOU race in nvidia-container-cli during container setup. Between the toolkit's security check of the container filesystem and its actual use, the attacker modifies the filesystem to create symlinks that escape the container mount namespace. This allows writing to arbitrary host paths or accessing the host's GPU device files with elevated privileges.WAF mitigation (if patching is not yet possible)
Add this rule to your WAF to block exploitation attempts while you schedule the patch.
WAF rules cannot mitigate container escape vulnerabilities. Defense-in-depth measures include:
- Restrict which container images can access GPU resources
- Use admission controllers (OPA/Gatekeeper) to block untrusted images from requesting GPU
- Enable seccomp and AppArmor profiles for GPU containers
- Run containers as non-root with minimal capabilities
- Monitor container runtime for filesystem escape attemptsHow to check if you are affected
- Check NVIDIA Container Toolkit version: nvidia-ctk --version — vulnerable if <= 1.16.1
- Check nvidia-container-runtime version: nvidia-container-runtime --version
- List containers with GPU access: docker ps --filter 'label=com.nvidia.volumes.needed'
- Check for suspicious symlinks in container layers: find /var/lib/docker/overlay2 -type l -name '*.so*' | head -20
- Monitor Docker daemon logs for container escape indicators: journalctl -u docker | grep -i 'nvidia\|escape\|mount'
- Verify container isolation: docker inspect --format '{{.HostConfig.Privileged}}' $(docker ps -q)
Indicators of compromise
- Containers creating symlinks to host filesystem paths during initialization
- Unexpected host file modifications originating from container processes
- nvidia-container-cli crashes or errors during container setup
- Containers accessing host GPU devices outside normal nvidia-smi paths
- File writes to /etc or /usr on the host from container UIDs
FAQ
Does this affect all Docker containers with GPU access?
Yes, any container using NVIDIA Container Toolkit 1.16.1 or earlier with default configuration is potentially vulnerable if an attacker can control the container image. This includes Docker, Podman, Kubernetes pods, and cloud GPU instances.
Is this exploitable in Kubernetes?
Yes. In Kubernetes, if an attacker can deploy a pod with GPU resource requests (nvidia.com/gpu), they can exploit this to escape to the host node. Admission controllers should restrict which workloads can request GPU resources.
Does this require a privileged container?
No. The vulnerability exists in the default, non-privileged container configuration. The TOCTOU race occurs during the NVIDIA container runtime setup before the container process starts, so standard container isolation does not prevent exploitation.
Are cloud GPU instances affected?
Yes. AWS, GCP, Azure, and other cloud providers offering GPU instances with NVIDIA Container Toolkit are affected if running vulnerable versions. Check with your cloud provider for patched AMIs/images.
How was this discovered?
This vulnerability was discovered by Wiz Research and responsibly disclosed to NVIDIA. The TOCTOU class of vulnerability is common in container runtimes that perform security checks before privileged operations.
Related Kernel CVEs
References
Track CVEs across your fleet automatically
Defensia scans your Linux servers and tells you exactly which ones are running vulnerable versions — including CVE-2024-0132. Free for 1 server.
Get started free