Basically the Agent automates the docker run command with everything that you need (this can become rather complex). You can see this in the third line of the console log:Executing: ['docker', 'run', '-t', '--gpus', 'all', '-l', 'clearml-worker-id=ip-172-31-28-179:0', '-l', 'clearml-parent-worker-id=ip-172-31-28-179:0', '-e', 'CLEARML_WORKER_ID=ip-172-31-28-179:0', '-e', 'CLEARML_DOCKER_IMAGE=nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04', '-e', 'CLEARML_TASK_ID=cdbfa9cda5ab4d86b012a87362026b23', '-v', '/tmp/.clearml_agent.bgu12h4e.cfg:/root/clearml.conf', '-v', '/tmp/clearml_agent.ssh.a991gp7v:/root/.ssh', '-v', '/home/ubuntu/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/home/ubuntu/.clearml/pip-cache:/root/.cache/pip', '-v', '/home/ubuntu/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/home/ubuntu/.clearml/cache:/clearml_agent_cache', '-v', '/home/ubuntu/.clearml/vcs-cache:/root/.clearml/vcs-cache', '--rm', 'nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04', 'bash', '-c', 'echo \'Binary::apt::APT::Keep-Downloaded-Packages "true";\' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL libsm6 libxext6 libxrender-dev libglib2.0-0" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip<20.2" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /root/clearml.conf /root/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring --id cdbfa9cda5ab4d86b012a87362026b23']
I can think of two solutions:
Fix local python environments and begin using virtual environments ( https://github.com/pyenv/pyenv for example) Use the agent in --docker mode. You won't need to worry about python versions but you will need to install Docker on that machine.
I even not sure what is the main problem. Am I right that the docker just can not find the version needed? If I get this error?
ERROR: Could not find a version that satisfies the requirement ipython==7.33.0 (from -r /tmp/cached-reqssiv6gjvc.txt (line 4)) (from versions: 0.10, 0.10.1, 0.10.2, 0.11, 0.12, 0.12.1, 0.13, 0.13.1, 0.13.2, 1.0.0, 1.1.0, 1.2.0, 1.2.1, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 3.0.0, 3.1.0, 3.2.0, 3.2.1, 3.2.2, 3.2.3, 4.0.0b1, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.1.0rc1, 4.1.0rc2, 4.1.0, 4.1.1, 4.1.2, 4.2.0, 4.2.1, 5.0.0b1, 5.0.0b2, 5.0.0b3, 5.0.0b4, 5.0.0rc1, 5.0.0, 5.1.0, 5.2.0, 5.2.1, 5.2.2, 5.3.0, 5.4.0, 5.4.1, 5.5.0, 5.6.0, 5.7.0, 5.8.0, 5.9.0, 5.10.0, 6.0.0rc1, 6.0.0, 6.1.0, 6.2.0, 6.2.1, 6.3.0, 6.3.1, 6.4.0, 6.5.0, 7.0.0b1, 7.0.0rc1, 7.0.0, 7.0.1, 7.1.0, 7.1.1, 7.2.0, 7.3.0, 7.4.0, 7.5.0, 7.6.0, 7.6.1, 7.7.0, 7.8.0, 7.9.0, 7.10.0, 7.10.1, 7.10.2, 7.11.0, 7.11.1, 7.12.0, 7.13.0, 7.14.0, 7.15.0, 7.16.0, 7.16.1, 7.16.2, 7.16.3)
ERROR: No matching distribution found for ipython==7.33.0 (from -r /tmp/cached-reqssiv6gjvc.txt (line 4))
Ok. I see this ‘nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04’
this is what clearml currently use?
I'm not personally familiar, but I'm sure searching for docker images with the python version you need will yield the required results 🙂
‘/home/ubuntu/.local/bin/clearml-agent daemon --docker’
the problem is that I use docker already. I am trying to run the code on the remote server which was previously ran in Colab. And I think that docker just can not find the appropriate version on the server… since I guess Colab used 3.8
In the UI, you can edit the docker image you want to use. You can then choose an image with the needed python pre-installed
This is the default image. I guess it doesn't have the python version you need to run with.
The json file looks like an error. Did your log download fail?
and where I can find the docker image that has the python version I need to run with?.. sorry 😳
I sent json file - it is not what you need?
OK. So now you have some task that you want to run with a specific docker image that you have in hand?
But I do use docker… I did not get your 2 point, please, clarify
In the "Execution" section go to "Container". You will need to edit the "image" section with the appropriate docker image. You will need to find a docker image that has the python version you need to run with.
oh.. I know nothing about this docker images… 😳 I just tried to follow instructions from the video and I do not fully understand what I am doing))
Did you run the agent with the --docker tag?
What version of python do you need to run on?
This is part of the log - I'll need the entire thing 🙂ERROR: Could not find a version that satisfies the requirement ipython==7.33.0 (from -r /tmp/cached-reqssiv6gjvc.txt (line 4)) (from versions: 0.10, 0.10.1, 0.10.2, 0.11, 0.12, 0.12.1, 0.13, 0.13.1, 0.13.2, 1.0.0, 1.1.0, 1.2.0, 1.2.1, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 3.0.0, 3.1.0, 3.2.0, 3.2.1, 3.2.2, 3.2.3, 4.0.0b1, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.1.0rc1, 4.1.0rc2, 4.1.0, 4.1.1, 4.1.2, 4.2.0, 4.2.1, 5.0.0b1, 5.0.0b2, 5.0.0b3, 5.0.0b4, 5.0.0rc1, 5.0.0, 5.1.0, 5.2.0, 5.2.1, 5.2.2, 5.3.0, 5.4.0, 5.4.1, 5.5.0, 5.6.0, 5.7.0, 5.8.0, 5.9.0, 5.10.0, 6.0.0rc1, 6.0.0, 6.1.0, 6.2.0, 6.2.1, 6.3.0, 6.3.1, 6.4.0, 6.5.0, 7.0.0b1, 7.0.0rc1, 7.0.0, 7.0.1, 7.1.0, 7.1.1, 7.2.0, 7.3.0, 7.4.0, 7.5.0, 7.6.0, 7.6.1, 7.7.0, 7.8.0, 7.9.0, 7.10.0, 7.10.1, 7.10.2, 7.11.0, 7.11.1, 7.12.0, 7.13.0, 7.14.0, 7.15.0, 7.16.0, 7.16.1, 7.16.2, 7.16.3) ERROR: No matching distribution found for ipython==7.33.0 (from -r /tmp/cached-reqssiv6gjvc.txt (line 4))
And also exactly what command line you used to run the agent?
You will need to find the appropriate docker image with the python version you're looking for.