I'll try to create a more classic image.
That is always better, though I remember we have some flag to allow that, you can try with:CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1 clearml-agent ...
Hi ShinyRabbit94 , can you verify the python version running in the agent (where requirements are not found) is the same as the one where you installed your requirements in the docker image?
Do you see the system_site_packages: true
in the configuration printed in the log (when the agent runs in the docker image)?
The logs continue like this :
` Summary - installed python packages:
pip:
- attrs==20.3.0
- backports.entry-points-selectable==1.1.1
- certifi==2021.10.8
- chardet==4.0.0
- clearml==1.1.4
- Cython==0.29.26
- distlib==0.3.4
- filelock==3.4.0
- furl==2.1.3
- future==0.18.2
- idna==2.10
- jsonschema==3.2.0
- numpy==1.21.5
- orderedmultidict==1.0.1
- pathlib2==2.3.6
- Pillow==8.4.0
- platformdirs==2.4.0
- psutil==5.8.0
- pyhocon==0.3.59
- PyJWT==2.0.1
- pyparsing==2.4.7
- pyrsistent==0.18.0
- python-dateutil==2.8.2
- PyYAML==5.4.1
- requests==2.25.1
- six==1.15.0
- typing==3.7.4.3
- urllib3==1.26.7
- virtualenv==20.10.0
Environment setup completed successfully `
I tried to fix the python binary in the config as well :agent.python_binary = /opt/venv/bin/python3
where :/opt/venv/bin/python3
is the output of which python
ran inside a docker container using my image.
In the clearml-agent
logs I see this :/root/.clearml/venvs-builds/3.8/bin/python -u /root/.clearml/venvs-builds/3.8/code/train.py
So I don't know if it's using the same python version or not.
/root/.clearml/venvs-builds/3.8/bin/python -u /root/.clearml/venvs-builds/3.8/code/train.py
This is the venv installed by the server (hopefully with system_site_packages: true
as you specified in the configuration file).
Can you exec into the image and do python -V
?
Well, the config is once thing, but if it can't find the python executable, it will use the same python version it's running from
Hi ShinyRabbit94
system_site_packages: true
This is set automatically when running in "docker mode" no need to worry 🙂
What is exactly the error you are getting ?
Could it be the container itself has the python packages installed in a venv not as "system packages" ?
Sorry for the late reply. It is indeed a venv, I though it would not be an issue since the PYTHONPATH
and the PATH
are both set to prioritize the venv. I'll try to create a more classic image.
I noticed logs start as follows :/usr/bin/python3.9 /usr/bin/python3.9: No module named pip /usr/local/bin/python3.8
even thought when starting the worker I see this :agent.python_binary = /opt/venv/bin/python3
If it helps, I tried changing the python version to 3.9 (which is also installed in my image). The change is reflected in the agent's config (the lines that appear when starting the worker) but it's still using 3.8 when executing the script.