RunPod - unsatisfied condition: cuda>=12.6

I noticed a problem where RunPod Serverless instance fails to start and keeps retrying until the 10-minute timeout, wasting both time and money.

start container for sprited/worker-comfyui:0.0.3: begin error starting container: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.6, please update your driver to a newer version, or use an earlier cuda container: unknown

x 100

The problem is that when this kind of error occurs, the request stays in the queue until it times out (defaults to 10 minute). There are two issues:

1. The job stays in the queue for full timeout, leaving the user waiting without any feedback.

2. Even though the job never starts, RunPod still charges per second (up to 10 minutes) even though nothing’s running.

The failure itself is straightforward: it tries to run CUDA 12.6 on a node with a CUDA driver lower than 12.6 (likely 12.4). The fix would be to downgrade your docker image to CUDA 12.4.

But the bigger problem seems systemic. If a node keeps failing to start, the RunPod orchestrator should detect this, mark the node as “bad,“ and route the request to another node. From my experience, this isn’t happening.

To be fair, there is 600-second timeout, but that’s still around 0.00031×600=18.6 cents (Nvidia 4090). More importantly, it locks up a node for 10 minutes wasting energy without doing useful work.

We could shorten the timeout, but sometimes inference really does take a long time. The key is to make sure timeouts only apply when the system is actually doing work.

Just wanted to share this observation. I think it’s a known issue, but hopefully it gets addressed sooner than later.

— Sprited Dev 🌱