Just to make sure, run the code on the machine itself to verify that python can actually detect the driver

Collecting pip<20.2
Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'pip'. No files were found to uninstall.

This one seems to be compatible: [nvcr.io/nvidia/pytorch:22.04-py3](http://nvcr.io/nvidia/pytorch:22.04-py3)

agent.cuda_version="11.2"

@<1523701070390366208:profile|CostlyOstrich36> I'm now running the agent with --docker , and I'm using task.create(docker="nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04")

to achieve running both the agent and the deployment on the same machine, adding --network=host to the run arguments solved it!

It means that there is an issue with the drivers. I suggest trying this docker image - nvcr.io/nvidia/pytorch:23.04-py3

Hi @<1734020162731905024:profile|RattyBluewhale45> , what version of pytorch are you specifying?

@<1523701070390366208:profile|CostlyOstrich36> do you have any ideas?

OK, then just try the docker image I suggested 🙂

Thank you

Hi @<1523701070390366208:profile|CostlyOstrich36> I am not specifying a version 🙂

I suggest running it in docker mode with a docker image that already has cuda installed

In the config file it should be something like this: agent.cuda_version="11.2" I think

I can install the correct torch version with this command:
pip install --pre torchvision --force-reinstall --index-url ` None ```

pip install --pre torchvision --force-reinstall --index-url None

This has been resolved now! Thank you for your help @<1523701070390366208:profile|CostlyOstrich36>

ERROR: This container was built for NVIDIA Driver Release 530.30 or later, but
       version 460.32.03 was detected and compatibility mode is UNAVAILABLE.

       [[System has unsupported display driver / cuda driver combination (CUDA_ERROR_SYSTEM_DRIVER_MISMATCH) cuInit()=803]]

I am running the agent with clearml-agent daemon --queue training

docker="nvidia/cuda:11.8.0-base-ubuntu20.04"

Isn't the problem that CUDA 12 is being installed?

Thank you I will try that

or cu11x

Thank you for getting back to me

But the process is still hanging, and not proceeding to actually running the clearml task

unrelated to the agent itself

I think it tries to get the latest one. Are you using the agent in docker mode? you can also control this via clearml.conf with agent.cuda_version

I have set agent{cuda_version: 11.2}

CUDA is the driver itself. The agent doesn't install CUDA but installs a compatible torch assuming that CUDA is properly installed.

What I dont understand is how to tell clearml to install this version of pytorch and torchvision, with cu118