Hi AgitatedDove14 , I’m talking about the following pip install.
After that pip install, it displays agent’s conf, shows installed packages, and launches the task (no installation)
` Running in Docker mode (v19.03 and above) - using default docker image: spoter ['-e CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1', '-e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1']
Running task '3ebb680b17874cda8dc7878ddf6fa735'
Storing stdout and stderr log to '/tmp/.clearml_agent_out.tsu2tddl.txt', '/tmp/.clearml_agent_out.tsu2tddl.txt'
Running Task 3ebb680b17874cda8dc7878ddf6fa735 inside docker: spoter arguments: []
Executing: ['docker', 'run', '-t', '--gpus', 'all', '-l', 'clearml-worker-id=Bottleneck:745823', '-l', 'clearml-parent-worker-id=Bottleneck:745823', '-e', 'CLEARML_WORKER_ID=Bottleneck:745823', '-e', 'CLEARML_DOCKER_IMAGE=spoter', '-e', 'CLEARML_TASK_ID=3ebb680b17874cda8dc7878ddf6fa735', '-v', '/home/gabi/.gitconfig:/root/.gitconfig', '-v', '/tmp/.clearml_agent._vwbx709.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/tmp/clearml_agent.ssh.s7m_pudm:/root/.ssh', '-v', '/home/gabi/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/home/gabi/.clearml/pip-cache:/root/.cache/pip', '-v', '/home/gabi/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/home/gabi/.clearml/cache:/clearml_agent_cache', '-v', '/home/gabi/.clearml/vcs-cache:/root/.clearml/vcs-cache', '-v', '/home/gabi/.clearml/venvs-cache:/root/.clearml/venvs-cache', '--rm', 'spoter', 'bash', '-c', 'echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip<20.2" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /tmp/clearml.conf ~/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring --id 3ebb680b17874cda8dc7878ddf6fa735']
Running Docker:
Executing: ['docker', 'run', '-t', '--gpus', 'all', '-l', 'clearml-worker-id=Bottleneck:745823', '-l', 'clearml-parent-worker-id=Bottleneck:745823', '-e', 'CLEARML_WORKER_ID=Bottleneck:745823', '-e', 'CLEARML_DOCKER_IMAGE=spoter', '-e', 'CLEARML_TASK_ID=3ebb680b17874cda8dc7878ddf6fa735', '-v', '/home/gabi/.gitconfig:/root/.gitconfig', '-v', '/tmp/.clearml_agent._vwbx709.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/tmp/clearml_agent.ssh.s7m_pudm:/root/.ssh', '-v', '/home/gabi/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/home/gabi/.clearml/pip-cache:/root/.cache/pip', '-v', '/home/gabi/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/home/gabi/.clearml/cache:/clearml_agent_cache', '-v', '/home/gabi/.clearml/vcs-cache:/root/.clearml/vcs-cache', '-v', '/home/gabi/.clearml/venvs-cache:/root/.clearml/venvs-cache', '--rm', 'spoter', 'bash', '-c', 'echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip<20.2" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /tmp/clearml.conf ~/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring --id 3ebb680b17874cda8dc7878ddf6fa735']

/opt/conda/bin/python3.7
pip 21.2.2 from /opt/conda/lib/python3.7/site-packages/pip (python 3.7)
Collecting pip<20.2
Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.2.2
Uninstalling pip-21.2.2:
Successfully uninstalled pip-21.2.2
Successfully installed pip-20.1.1 `

Now I’m just wondering if I could remove the PIP install at the very beginning, so it starts straightaway

AbruptCow41 CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1 does exactly that 🙂 BTW, I would just set the venv cache and this means it will just be able to restore the entire thing (even if you have changed the requirements
https://github.com/allegroai/clearml-agent/blob/077148be00ead21084d63a14bf89d13d049cf7db/docs/clearml.conf#L115

the problem was docker, that had as entrypoint a bash script with python train.py --epochs=300 hardcoded, so I guess it was never acutally running the task setup from clearml.

Can you add a small code snippet that reproduces this?

So are you saying why do we need to install a specific pip version ?
You can "disable it" by selecting a very high version
pip_version: "<40"https://github.com/allegroai/clearml-agent/blob/077148be00ead21084d63a14bf89d13d049cf7db/docs/clearml.conf#L67

Thanks for your answer, but it didn’t work 😕

before the repo was already in the docker, but now it is running the agent inside the docker (so setting a virtualenv, and cloning the repo, and installing the packages)

Yes, actually ensuring pip is there cannot be skipped (I think in the past it cased to many issues, hence the version limit etc.)
Are you saying it takes a lot of time when running? How long is the actual process that the Task is running (just to normalize times here)

so I removed the entrypoint, and now I can see that it tries to install the packages, but it fails because it can’t download the repo

right, but it’d still try to install it

not that much, I was just wondering if it was possible :-)

You have to define it, but this can be done really quickly under the task, adding
task.set_base_docker(docker_image="your docker image", docker_arguments='-e CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1')for example, or you can also add it to the agent’s configuration if you want to set this always with agent.default_docker.arguments https://clear.ml/docs/latest/docs/configs/clearml_conf/#agentdefault_docker

Hi AbruptCow41 ,

also, is there a way for it to not install the requirements, and simply run the task?

Yes, you can specify the env you like to use and disable the requirements installation:
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL env var to define the python environment you are working with
CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL to skip the installations

how do I mount my local ssh folder into

/root/.ssh/

docker when running clearml-agent?

this should be done by default. you should be able to see it in the docker command (at the top of the task’s console output)

Thanks TimelyPenguin76 for your answer! So indeed it was mounting it, and how do I check that “CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL” is working in my agent in docker?

how do I mount my local ssh folder into /root/.ssh/ docker when running clearml-agent?
also, is there a way for it to not install the requirements, and simply run the task?

Also, what version of clearml & clearml-agent are you using?

great! and I saw that there were some system packages needed for opencv that were installed automatically that could be turned off. Now I’m just wondering if I could remove the PIP install at the very beginning, so it starts straightaway

Hi AbruptCow41 ,

I think you need to call Task.init before creating the argparser args