Guides

Why your GPU container breaks: CUDA, drivers and pinned tags

Renting a GPU is easy. Getting a working stack on it is where people lose an evening, because four things have to agree — the host driver, the CUDA toolkit, the framework build and the card’s architecture — and nothing tells you which one is wrong. This is the short version of what actually goes wrong, written while wiring GPU support ourselves.

The one sentence to remember

The driver belongs to the host; the toolkit belongs to your image. You can change the second and never the first. Almost every confusing CUDA error is that sentence being violated.

1. Toolkit newer than the driver

The classic:

CUDA driver version is insufficient for CUDA runtime version

Your image ships a CUDA toolkit newer than the host’s driver supports. Inside a container you cannot fix this: the driver is the host’s, and apt install nvidia-driver-… in a container either fails or, worse, appears to succeed and changes nothing.

The fix is to run an image built against an older CUDA. Drivers are backward compatible with older toolkits, never forward compatible with newer ones.

# What the HOST driver supports — the ceiling you cannot raise
nvidia-smi | head -4

# What your image actually contains
nvcc --version
python -c "import torch; print(torch.version.cuda, torch.cuda.is_available())"

2. The card is newer than the build

This one is nastier, because it only appears when you change card. Every GPU architecture has a compute capability, and a build supports only the ones it was compiled for:

no kernel image is available for execution on the device
# or
sm_120 is not compatible with the current PyTorch installation

A CUDA 12.1 image is fine on Ampere and Ada — L4, L40S, A100 — and will not run on Blackwell cards such as the RTX 5090, which need CUDA 12.8 or newer. The trap is that the pod starts, and starts billing, and only then fails when something first touches the GPU.

So the card and the image have to be checked against each other before launch, not after. In our own implementation that is a filter which refuses to offer a workload on a card its CUDA build predates, for exactly this reason.

3. The tag moved under you

“It worked last week” is usually this. A floating tag like :latest or :cuda is a moving target, and GPU images move often — a rebuilt image can bring a new PyTorch, a new CUDA, or a custom node that no longer matches.

Pin to a tag that names its versions, and prefer projects that publish such tags:

# Moving target — the same command, a different stack, any given week
ghcr.io/ai-dock/comfyui:latest-cuda

# Pinned: CUDA version and release both stated
ghcr.io/ai-dock/comfyui:v2-cuda-12.1.1-base-22.04-v0.2.7

# Strongest: a digest cannot change at all
ghcr.io/ai-dock/comfyui@sha256:...

We pin the images we run and treat the tag as part of the configuration, not an afterthought. It is the single cheapest thing you can do to stop losing evenings.

4. The image is the wrong shape entirely

Worth checking before you debug anything: some images that look right are built for a different job. RunPod’s own worker-comfyui is the official ComfyUI image and it is a serverless worker — it exposes /run and /runsync and never serves the node-graph interface most people are actually after.

Neither image is wrong; they answer different questions. But “official” is not the same as “the one you want”, and an hour spent wondering why the UI will not load is an hour spent on the wrong problem.

5. What a rented GPU container cannot do

A pod is a container on someone else’s host, which rules some things out no matter how the image is built. The one that surprises people most:

mknod: /dev/net/tun: Operation not permitted

Providers including RunPod do not map /dev/net/tun into pods, and CAP_NET_ADMIN alone is not sufficient — the device has to be mapped too. So WireGuard, Tailscale, OpenVPN and every other kernel-mode VPN simply will not start inside one.

Two ways round it. Tailscale runs in --tun=userspace-networking mode without the device, at the cost of rewriting inbound source IPs. Or skip the VPN: if what you need is a port from the pod on another machine, an SSH forward does exactly that with no capabilities at all.

# Publish the pod's port on your own server's loopback.
# -N no command, and ExitOnForwardFailure so a failed forward exits
# instead of leaving a connection that looks healthy and forwards nothing.
ssh -N -o ExitOnForwardFailure=yes \
    -p <mapped-ssh-port> \
    -L 127.0.0.1:8188:127.0.0.1:8188 \
    root@<pod-ip>

That is the approach we settled on, having started out intending to use WireGuard and found it impossible.

6. Do not publish the port

A GPU pod gets a public IP, and the ports you expose are reachable by anyone who finds them. That matters more than usual here, because these tools generally have no authentication at all. ComfyUI has none, and it runs arbitrary Python through custom nodes — an exposed instance is remote code execution attached to a GPU somebody is paying for.

Check the default. RunPod’s is 8888/http,22/tcp, so a pod created without thinking about ports publishes 8888 to the internet. Expose SSH only, reach the tool through the tunnel, and put a login in front of it on the machine you control.

The five-minute checklist

  1. Check the host driver first — nvidia-smi. It is the ceiling.
  2. Match the image to the card, not just to the workload. Newer silicon needs a newer CUDA.
  3. Pin the tag, ideally by digest.
  4. Confirm the image is the shape you want — a serverless worker is not a UI.
  5. Expose nothing but SSH, and put a login in front of anything you tunnel out.
  6. Set a hard stop. A forgotten GPU at $2–4/hour is $1,500–2,900 a month.

Related