Thanks for responding, but it seems we have fixed the issue. The file structure for a K8s mongo and ES are slightly different which we hadn’t noticed!

So running this train script will sometimes work and sometimes give the error of the original post

Thanks for responding. Here you can find some references:

The runner is a ubuntu machine on which a specific user is made. Here we have a venv from which we run:

clearml-agent daemon --queue gpu_12gb --detached --gpus 1
clearml-agent daemon --queue gpu_24gb --detached --gpus 0

clearml-agent daemon --queue no_gpu --detached

This user has a clearml.conf file in its home directory. When I run clearml-data commands as this user from the venv everything works as expected.

I a...

I am training using the tools/train.py script, must I add the snippet here then?

I’ll try that out, thank you!