Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello, I Am Trying To Run The

Hello,

I am trying to run the clearml-agent in docker mode. I use this command to start it :
sudo clearml-agent daemon --cpu-only --queue training_queue --docker trainerthen this command :
clearml-task --project train --name train2 --script train.py --requirements requirements.txt --queue training_queueto enqueue a task.

Now, my docker image trainer already has all the requirements needed to run my task, so ideally, I would like to skip the requirements installation.
I thought I could do so by setting :
# virtual environment inheres packages from system system_site_packages: truebut it does seem to work (requirements not found when the script is executed.
Have I misunderstood the parameter ? If so is there a way to do what I described, or shall all the requirements my script need be specified in the requirements.txt ?

  
  
Posted 2 years ago
Votes Newest

Answers 16


If it helps, I tried changing the python version to 3.9 (which is also installed in my image). The change is reflected in the agent's config (the lines that appear when starting the worker) but it's still using 3.8 when executing the script.

  
  
Posted 2 years ago

I'll try to create a more classic image.

That is always better, though I remember we have some flag to allow that, you can try with:
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1 clearml-agent ...

  
  
Posted 2 years ago

Hi ShinyRabbit94 , can you verify the python version running in the agent (where requirements are not found) is the same as the one where you installed your requirements in the docker image?

  
  
Posted 2 years ago

Do you see the system_site_packages: true in the configuration printed in the log (when the agent runs in the docker image)?

  
  
Posted 2 years ago

Sorry for the late reply. It is indeed a venv, I though it would not be an issue since the PYTHONPATH and the PATH are both set to prioritize the venv. I'll try to create a more classic image.

  
  
Posted 2 years ago

The logs continue like this :
` Summary - installed python packages:
pip:

  • attrs==20.3.0
  • backports.entry-points-selectable==1.1.1
  • certifi==2021.10.8
  • chardet==4.0.0
  • clearml==1.1.4
  • Cython==0.29.26
  • distlib==0.3.4
  • filelock==3.4.0
  • furl==2.1.3
  • future==0.18.2
  • idna==2.10
  • jsonschema==3.2.0
  • numpy==1.21.5
  • orderedmultidict==1.0.1
  • pathlib2==2.3.6
  • Pillow==8.4.0
  • platformdirs==2.4.0
  • psutil==5.8.0
  • pyhocon==0.3.59
  • PyJWT==2.0.1
  • pyparsing==2.4.7
  • pyrsistent==0.18.0
  • python-dateutil==2.8.2
  • PyYAML==5.4.1
  • requests==2.25.1
  • six==1.15.0
  • typing==3.7.4.3
  • urllib3==1.26.7
  • virtualenv==20.10.0
    Environment setup completed successfully `
  
  
Posted 2 years ago

I tried to fix the python binary in the config as well :
agent.python_binary = /opt/venv/bin/python3where :
/opt/venv/bin/python3is the output of which python ran inside a docker container using my image.
In the clearml-agent logs I see this :
/root/.clearml/venvs-builds/3.8/bin/python -u /root/.clearml/venvs-builds/3.8/code/train.pySo I don't know if it's using the same python version or not.

  
  
Posted 2 years ago

/root/.clearml/venvs-builds/3.8/bin/python -u /root/.clearml/venvs-builds/3.8/code/train.pyThis is the venv installed by the server (hopefully with system_site_packages: true as you specified in the configuration file).
Can you exec into the image and do python -V ?

  
  
Posted 2 years ago

Well, the config is once thing, but if it can't find the python executable, it will use the same python version it's running from

  
  
Posted 2 years ago

yes, it's set to true in the logs as well

  
  
Posted 2 years ago

Python 3.8.12

  
  
Posted 2 years ago

I noticed logs start as follows :
/usr/bin/python3.9 /usr/bin/python3.9: No module named pip /usr/local/bin/python3.8

  
  
Posted 2 years ago

and agent.python_binary is empty.

  
  
Posted 2 years ago

(in the logs)

  
  
Posted 2 years ago

Hi ShinyRabbit94

system_site_packages: true

This is set automatically when running in "docker mode" no need to worry 🙂
What is exactly the error you are getting ?
Could it be the container itself has the python packages installed in a venv not as "system packages" ?

  
  
Posted 2 years ago

even thought when starting the worker I see this :
agent.python_binary = /opt/venv/bin/python3

  
  
Posted 2 years ago
591 Views
16 Answers
2 years ago
one year ago
Tags
Similar posts