Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi! I’M Running An Experiment As Follows:

Hi! I’m running an experiment as follows:
create argparser -> args Task.init (default args) call train(args)Then I clone it and change the parameters in the UI, and run it with clearml-agent but when calling train(args) it is still using the same parameters as when the task was created…is this the expected behaviour?

  
  
Posted one year ago
Votes Newest

Answers 18


Now I’m just wondering if I could remove the PIP install at the very beginning, so it starts straightaway

AbruptCow41 CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1 does exactly that 🙂 BTW, I would just set the venv cache and this means it will just be able to restore the entire thing (even if you have changed the requirements
https://github.com/allegroai/clearml-agent/blob/077148be00ead21084d63a14bf89d13d049cf7db/docs/clearml.conf#L115

  
  
Posted one year ago

right, but it’d still try to install it

  
  
Posted one year ago

Hi AgitatedDove14 , I’m talking about the following pip install.
After that pip install, it displays agent’s conf, shows installed packages, and launches the task (no installation)
` Running in Docker mode (v19.03 and above) - using default docker image: spoter ['-e CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1', '-e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1']
Running task '3ebb680b17874cda8dc7878ddf6fa735'
Storing stdout and stderr log to '/tmp/.clearml_agent_out.tsu2tddl.txt', '/tmp/.clearml_agent_out.tsu2tddl.txt'
Running Task 3ebb680b17874cda8dc7878ddf6fa735 inside docker: spoter arguments: []
Executing: ['docker', 'run', '-t', '--gpus', 'all', '-l', 'clearml-worker-id=Bottleneck:745823', '-l', 'clearml-parent-worker-id=Bottleneck:745823', '-e', 'CLEARML_WORKER_ID=Bottleneck:745823', '-e', 'CLEARML_DOCKER_IMAGE=spoter', '-e', 'CLEARML_TASK_ID=3ebb680b17874cda8dc7878ddf6fa735', '-v', '/home/gabi/.gitconfig:/root/.gitconfig', '-v', '/tmp/.clearml_agent._vwbx709.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/tmp/clearml_agent.ssh.s7m_pudm:/root/.ssh', '-v', '/home/gabi/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/home/gabi/.clearml/pip-cache:/root/.cache/pip', '-v', '/home/gabi/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/home/gabi/.clearml/cache:/clearml_agent_cache', '-v', '/home/gabi/.clearml/vcs-cache:/root/.clearml/vcs-cache', '-v', '/home/gabi/.clearml/venvs-cache:/root/.clearml/venvs-cache', '--rm', 'spoter', 'bash', '-c', 'echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip<20.2" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /tmp/clearml.conf ~/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring --id 3ebb680b17874cda8dc7878ddf6fa735']
Running Docker:
Executing: ['docker', 'run', '-t', '--gpus', 'all', '-l', 'clearml-worker-id=Bottleneck:745823', '-l', 'clearml-parent-worker-id=Bottleneck:745823', '-e', 'CLEARML_WORKER_ID=Bottleneck:745823', '-e', 'CLEARML_DOCKER_IMAGE=spoter', '-e', 'CLEARML_TASK_ID=3ebb680b17874cda8dc7878ddf6fa735', '-v', '/home/gabi/.gitconfig:/root/.gitconfig', '-v', '/tmp/.clearml_agent._vwbx709.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/tmp/clearml_agent.ssh.s7m_pudm:/root/.ssh', '-v', '/home/gabi/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/home/gabi/.clearml/pip-cache:/root/.cache/pip', '-v', '/home/gabi/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/home/gabi/.clearml/cache:/clearml_agent_cache', '-v', '/home/gabi/.clearml/vcs-cache:/root/.clearml/vcs-cache', '-v', '/home/gabi/.clearml/venvs-cache:/root/.clearml/venvs-cache', '--rm', 'spoter', 'bash', '-c', 'echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip<20.2" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /tmp/clearml.conf ~/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring --id 3ebb680b17874cda8dc7878ddf6fa735']

/opt/conda/bin/python3.7
pip 21.2.2 from /opt/conda/lib/python3.7/site-packages/pip (python 3.7)
Collecting pip<20.2
Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.2.2
Uninstalling pip-21.2.2:
Successfully uninstalled pip-21.2.2
Successfully installed pip-20.1.1 `

  
  
Posted one year ago

Hi AbruptCow41 ,

also, is there a way for it to not install the requirements, and simply run the task?

Yes, you can specify the env you like to use and disable the requirements installation:
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL env var to define the python environment you are working with
CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL to skip the installations

how do I mount my local ssh folder into

/root/.ssh/

docker when running clearml-agent?

this should be done by default. you should be able to see it in the docker command (at the top of the task’s console output)

  
  
Posted one year ago

So are you saying why do we need to install a specific pip version ?
You can "disable it" by selecting a very high version
pip_version: "<40"https://github.com/allegroai/clearml-agent/blob/077148be00ead21084d63a14bf89d13d049cf7db/docs/clearml.conf#L67

  
  
Posted one year ago

before the repo was already in the docker, but now it is running the agent inside the docker (so setting a virtualenv, and cloning the repo, and installing the packages)

  
  
Posted one year ago

the problem was docker, that had as entrypoint a bash script with python train.py --epochs=300 hardcoded, so I guess it was never acutally running the task setup from clearml.

  
  
Posted one year ago

so I removed the entrypoint, and now I can see that it tries to install the packages, but it fails because it can’t download the repo

  
  
Posted one year ago

great! and I saw that there were some system packages needed for opencv that were installed automatically that could be turned off. Now I’m just wondering if I could remove the PIP install at the very beginning, so it starts straightaway

  
  
Posted one year ago

not that much, I was just wondering if it was possible :-)

  
  
Posted one year ago

Can you add a small code snippet that reproduces this?

  
  
Posted one year ago

You have to define it, but this can be done really quickly under the task, adding
task.set_base_docker(docker_image="your docker image", docker_arguments='-e CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1')for example, or you can also add it to the agent’s configuration if you want to set this always with agent.default_docker.arguments https://clear.ml/docs/latest/docs/configs/clearml_conf/#agentdefault_docker

  
  
Posted one year ago

Thanks TimelyPenguin76 for your answer! So indeed it was mounting it, and how do I check that “CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL” is working in my agent in docker?

  
  
Posted one year ago

Also, what version of clearml & clearml-agent are you using?

  
  
Posted one year ago

Thanks for your answer, but it didn’t work 😕

  
  
Posted one year ago

Yes, actually ensuring pip is there cannot be skipped (I think in the past it cased to many issues, hence the version limit etc.)
Are you saying it takes a lot of time when running? How long is the actual process that the Task is running (just to normalize times here)

  
  
Posted one year ago

Hi AbruptCow41 ,

I think you need to call Task.init before creating the argparser args

  
  
Posted one year ago

how do I mount my local ssh folder into /root/.ssh/ docker when running clearml-agent?
also, is there a way for it to not install the requirements, and simply run the task?

  
  
Posted one year ago