Server that I'm using has GPU Driver Version: 455.23.05 and CUDA Version: 11.1, Conda is also installed and clearml-serving is installing cuda version 10.1 for which gpu drivers should be >= 418.39 so I guess version mismatch is not the problem and currently I can't update gpu drivers since other processes are running.

And also I tried overriding clearml.conf file and changed default docker image by modifying below line
image: "nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04"to this:
image: "nvidia/cuda:11.1-cudnn8-runtime-ubuntu18.04"But still same same error the provided PTX was compiled with an unsupported toolchain occurred while launching triton-engine