great! and I saw that there were some system packages needed for opencv that were installed automatically that could be turned off. Now I’m just wondering if I could remove the PIP install at the very beginning, so it starts straightaway
Hi AgitatedDove14 , I’m talking about the following pip install.
After that pip install, it displays agent’s conf, shows installed packages, and launches the task (no installation)
` Running in Docker mode (v19.03 and above) - using default docker image: spoter ['-e CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1', '-e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1']
Running task '3ebb680b17874cda8dc7878ddf6fa735'
Storing stdout and stderr log to '/tmp/.clearml_agent_out.tsu2tddl.txt', '/tmp/.clearml_agent_out.tsu2tddl.txt'
Running Task 3ebb680b17874cda8dc7878ddf6fa735 inside docker: spoter arguments: []
Executing: ['docker', 'run', '-t', '--gpus', 'all', '-l', 'clearml-worker-id=Bottleneck:745823', '-l', 'clearml-parent-worker-id=Bottleneck:745823', '-e', 'CLEARML_WORKER_ID=Bottleneck:745823', '-e', 'CLEARML_DOCKER_IMAGE=spoter', '-e', 'CLEARML_TASK_ID=3ebb680b17874cda8dc7878ddf6fa735', '-v', '/home/gabi/.gitconfig:/root/.gitconfig', '-v', '/tmp/.clearml_agent._vwbx709.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/tmp/clearml_agent.ssh.s7m_pudm:/root/.ssh', '-v', '/home/gabi/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/home/gabi/.clearml/pip-cache:/root/.cache/pip', '-v', '/home/gabi/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/home/gabi/.clearml/cache:/clearml_agent_cache', '-v', '/home/gabi/.clearml/vcs-cache:/root/.clearml/vcs-cache', '-v', '/home/gabi/.clearml/venvs-cache:/root/.clearml/venvs-cache', '--rm', 'spoter', 'bash', '-c', 'echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip<20.2" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /tmp/clearml.conf ~/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring --id 3ebb680b17874cda8dc7878ddf6fa735']
Running Docker:
Executing: ['docker', 'run', '-t', '--gpus', 'all', '-l', 'clearml-worker-id=Bottleneck:745823', '-l', 'clearml-parent-worker-id=Bottleneck:745823', '-e', 'CLEARML_WORKER_ID=Bottleneck:745823', '-e', 'CLEARML_DOCKER_IMAGE=spoter', '-e', 'CLEARML_TASK_ID=3ebb680b17874cda8dc7878ddf6fa735', '-v', '/home/gabi/.gitconfig:/root/.gitconfig', '-v', '/tmp/.clearml_agent._vwbx709.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/tmp/clearml_agent.ssh.s7m_pudm:/root/.ssh', '-v', '/home/gabi/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/home/gabi/.clearml/pip-cache:/root/.cache/pip', '-v', '/home/gabi/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/home/gabi/.clearml/cache:/clearml_agent_cache', '-v', '/home/gabi/.clearml/vcs-cache:/root/.clearml/vcs-cache', '-v', '/home/gabi/.clearml/venvs-cache:/root/.clearml/venvs-cache', '--rm', 'spoter', 'bash', '-c', 'echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip<20.2" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /tmp/clearml.conf ~/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring --id 3ebb680b17874cda8dc7878ddf6fa735']
/opt/conda/bin/python3.7
pip 21.2.2 from /opt/conda/lib/python3.7/site-packages/pip (python 3.7)
Collecting pip<20.2
Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.2.2
Uninstalling pip-21.2.2:
Successfully uninstalled pip-21.2.2
Successfully installed pip-20.1.1 `
Can you add a small code snippet that reproduces this?
Now I’m just wondering if I could remove the PIP install at the very beginning, so it starts straightaway
AbruptCow41 CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1
does exactly that 🙂 BTW, I would just set the venv cache and this means it will just be able to restore the entire thing (even if you have changed the requirements
https://github.com/allegroai/clearml-agent/blob/077148be00ead21084d63a14bf89d13d049cf7db/docs/clearml.conf#L115
Also, what version of clearml
& clearml-agent
are you using?
the problem was docker, that had as entrypoint a bash script with python train.py --epochs=300
hardcoded, so I guess it was never acutally running the task setup from clearml.
before the repo was already in the docker, but now it is running the agent inside the docker (so setting a virtualenv, and cloning the repo, and installing the packages)
So are you saying why do we need to install a specific pip version ?
You can "disable it" by selecting a very high versionpip_version: "<40"
https://github.com/allegroai/clearml-agent/blob/077148be00ead21084d63a14bf89d13d049cf7db/docs/clearml.conf#L67
how do I mount my local ssh folder into /root/.ssh/
docker when running clearml-agent?
also, is there a way for it to not install the requirements, and simply run the task?
so I removed the entrypoint, and now I can see that it tries to install the packages, but it fails because it can’t download the repo
not that much, I was just wondering if it was possible :-)
Yes, actually ensuring pip is there cannot be skipped (I think in the past it cased to many issues, hence the version limit etc.)
Are you saying it takes a lot of time when running? How long is the actual process that the Task is running (just to normalize times here)
Thanks TimelyPenguin76 for your answer! So indeed it was mounting it, and how do I check that “CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL” is working in my agent in docker?
Thanks for your answer, but it didn’t work 😕
Hi AbruptCow41 ,
also, is there a way for it to not install the requirements, and simply run the task?
Yes, you can specify the env you like to use and disable the requirements installation:CLEARML_AGENT_SKIP_PIP_VENV_INSTALL
env var to define the python environment you are working withCLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL
to skip the installations
how do I mount my local ssh folder into
/root/.ssh/
docker when running clearml-agent?
this should be done by default. you should be able to see it in the docker command (at the top of the task’s console output)
You have to define it, but this can be done really quickly under the task, addingtask.set_base_docker(docker_image="your docker image", docker_arguments='-e CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1')
for example, or you can also add it to the agent’s configuration if you want to set this always with agent.default_docker.arguments
https://clear.ml/docs/latest/docs/configs/clearml_conf/#agentdefault_docker
Hi AbruptCow41 ,
I think you need to call Task.init
before creating the argparser args