I think you are correct with your guess that the services were not shut down properly. I noticed that some services were still shown as running on the clear ml dashboard. I aborted all and at least got rid of the error ValueError: triton-server process ended with error code 1 . But the two errors you named are still there and I also got these two warnings:
clearml-serving-triton | Warning: more than one valid Controller Tasks found, using Task ID=4709b0b383a04bb1a033e99fd325dcbf
clearml-serving-triton | WARNING: [Torch-TensorRT] - Unable to read CUDA capable devices. Return status: 35
And the first warning ist definetly right! I started a lot of service controllers by calling clearml-serving create --name "serving example" as I thought I would need to, to get a fresh startup. And honestly I do not know how to shut them down. Even after restarting my computer they are still running. Could this be a thing? Do you have a solution for this warning?
Again to your advice: I am wondering why the gpu is tried to be allocated as I trained the model on the cpu and used the 'docker-compose-triton.yml' (not the http://docker-compose-triton-gpu.to start the docker container. Nevertheless my machine has a gpu installed which I used for some testing, so it is working in general but I am not sure about the docker container.