@<1523701070390366208:profile|CostlyOstrich36> Is there a way to tell clearml to not try to detect the Installed package ?

From the environment variable

Oh, so you mean CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/path/to/my/venv/bin/python3 ??

Hi @<1576381444509405184:profile|ManiacalLizard2> , I think this is the env var you're looking for
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL
None

So I tried:

CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/data/hieu/opt/python-venv/fastai/bin/python3.10
clearml-agent  daemon  --queue no_venv

Then enqueue a cloned task to no_venv
It is still trying to create a venv (and fail):

[...]
tag = 
docker_cmd = 
entry_point = debug.py
working_dir = apple_ic
created virtual environment CPython3.10.10.final.0-64 in 140ms
  creator CPython3Posix(dest=/data/hieu/deleteme/clearml-agent/venvs-builds/3.10, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mhtrinh/.local/share/virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.0.0, wheel==0.40.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Using cached repository in "/home/mhtrinh/.clearml/vcs-cache/KitchenSink.44ea8add86cace4fd712971ff93b965c/KitchenSink"
[...]
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.2.1
    Uninstalling pip-23.2.1:
      Successfully uninstalled pip-23.2.1
Successfully installed pip-22.2.2
Collecting Cython
  Using cached Cython-3.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
Installing collected packages: Cython
Successfully installed Cython-3.0.0
Adding venv into cache: /home/mhtrinh/.clearml/venvs-builds/3.10
Running task id [4a413e955b6e47c38939e0b1fe5a8786]:
[apple_ic]$ /home/mhtrinh/.clearml/venvs-builds/3.10/bin/python -u debug.py
Summary - installed python packages:
pip:
- Cython==3.0.0
Environment setup completed successfully
Starting Task Execution:
Traceback (most recent call last):
  File "/data/hieu/deleteme/clearml-agent/venvs-builds/3.10/task_repository/KitchenSink/apple_ic/debug.py", line 2, in <module>
    import livsdk.livbatch
ModuleNotFoundError: No module named 'livsdk'
2023-08-14 20:38:06
Process failed, exit code 1

oh, looks like I need to empty the Installed Package before enqueue the cloned task

Set that env var in the terminal before running the agent ?

Yes

no, it's an environment variable

Should I put that in the clearml.conf file?

Yes

Found a trick to have empty Installed package:
clearml.Task.force_requirements_env_freeze(force=True,requirements_file="/dev/null")
Not sure if this is the right way or not ...

you should set it on the machine running the agent

But then how did the agent know where is the venv that it needs to use?

So I now just need to find a way to not populate Installed Package in the first place