so now there is the user conflict between the host and the agent inside the container

` if name == "main":

task = Task.init(project_name="dummy",
             task_name="pretraining",
             task_type=Task.TaskTypes.training,
             reuse_last_task_id=False)

task.connect(args)
print('Arguments: {}'.format(args))

# only create the task, we will actually execute it later
task.execute_remotely()

spawn_dist.run(args) `I added it to this script and use it as a starting point   https://github.com/facebookresearch/fastMRI/bl...

I'm running now the the code shown above and will let you know if there is still an issue

using this code in https://github.com/facebookresearch/fastMRI/blob/master/banding_removal/scripts/pretrain.py
` if name == "main":

task = Task.init(project_name="dummy",
             task_name="pretraining",
             task_type=Task.TaskTypes.training,
             reuse_last_task_id=False)

task.connect(args)
print('Arguments: {}'.format(args))

# only create the task, we will actually execute it later
task.execute_remotely()

spawn_dist.run...

This happens inside the agent, since I use task.execute_remotely() I guess. The agent runs on ubuntu 18.04 and not in docker mode

curl -XPUT -H 'Content-Type: application/json' 'localhost:9200/_settings' -d '{"index" : {"number_of_replicas" : 0}}This command made all my indices beside the broken one which is still red, come green again. It comes from https://stackoverflow.com/questions/63403972/elasticsearch-index-in-red-health/63405623#63405623 .

My code produces now an error inside one of the threads, but that should be an issue on my side. Still this issue inside a child thread was not detected as failure and the training task resulted in "completed". This error happens now with the Task.init inside the if __name__ == "__main__": as seen above in the code snippet.

clearml-agent daemon --gpus 0 --queue default --docker nvidia/cuda:11.3.0-cudnn8-runtime-ubuntu18.0 causes not using the GPUs because of missing libs.

Hi AgitatedDove14 one more question about efficient caching, is it possible to cache/share docker images between agents?

Ok it is more a docker issue, I guess it is not feasible reading the thread.

SuccessfulKoala55 do you have any example? I guess a lot of people face this issue

Exactly, all agents should share the cache that is mounted via nfs. I think it is working now 🙂

tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64

no it only needs the requirements.txt of project-a if I run a task of project-a

` # pip cache folder mapped into docker, used for python package caching
docker_pip_cache = /clearml-cache/pip-cache
# apt cache folder mapped into docker, used for ubuntu package caching
docker_apt_cache = /clearml-cache/apt-cache

docker_internal_mounts {
     apt_cache: "/clearml-cache/apt-cache"
     pip_cache: "/clearml-cache/pip-cache"
     vcs_cache: "/clearml-cache/vcs-cache"
     venv_build: "/clearml-cache/venvs-builds"
     pip_download: "/cl...

docker run --gpus device=0 --rm -it nvidia/cuda:11.3.0-cudnn8-runtime-ubuntu18.04 bash worked, I could run in it nvidia-smi and see gpu 0

thanks, that seems to be at least a solution

SuccessfulKoala55 Hey, for us artifact download urls, model download urls, images in plots and debug image urls are broken. In the linked example I can see a solution for the debug images and potentially plot images but cant find the artifacts and model urls inside ES. Are those urls maybe stored inside the mongodb? Any idea where to find them?

thanks a lot, yes it was the daemon :man-facepalming: I already could recover one checkpoint!

thanks for the info, thats really bad 😬 I thought that the output_uri defaults to the fileserver 🙄

but this would be still part of the clearml.conf right? I would prefer a way around clearml.conf to avoid resolving the variables

Try to restart ES and see if it helps

docker-compose down / up does not help

Hey Natan, good point! But I have actually set both

the error your are citing happens when running clearml-agent daemon --gpus 0 --queue default --docker nvidia/cuda

I also saw it a few times

We run a lot of pipelines that are cpu only with some parallel steps. Its just about improving the execution time

using top inside the elasticsearch container shows elastic+ 20  0  17.0g  8.7g 187584 S  2.3 27.2  1:09.18 java that the 8g are reserved. So setting ES_JAVA_OPTS: -Xms8g -Xmx8g should work.

I will try to recover it, but anyway the learning is to fully separate the fileserver and any output location from mongo, redis and elastic. Also maybe it makes sense the improve the ES setup to have replicas

SuccessfulKoala55 I'm currently inside the docker container to recover the ckpt files. But /root/.clearml/venvs-builds seems to be empty. Any idea where I could then find the ckpt files?

So I don't need docker_internal_mounts at all?