ThoughtfulBadger56 Have you uncommented the existing venvs_cache section in the config file?
https://clear.ml/docs/latest/docs/clearml_agent#virtual-environment-reuse

I have install a python environment by virtualenv tool, let's say

/home/frank/env

and python is

/home/frank/env/bin/python3.

How to reuse the virtualenv by setting clearml agent?

So the agent is already caching the entire venv for you, nothing to worry about, just make sure you have this line in clearml:
https://github.com/allegroai/clearml-agent/blob/249b51a31bee97d63f41c6d5542e657962008b68/docs/clearml.conf#L131
No need to provide it an existing venv, if you are running in docker mode, you can provide
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/home/frank/env/bin/python3

Of course I did.

Even I only set agent.venvs_cache.path=/home/frank/env in the clearml.conf, the train still failed to start. ClearML always create a new venvs and install Cython and some other packages, so weird!
... Installing collected packages: Cython Successfully installed Cython-0.29.32 Adding venv into cache: /general-user/frank/.clearml/venvs-builds/3.8 Running task id [42a050853c43445ebf9248bd4aa54091]: [.]$ /general-user/frank/.clearml/venvs-builds/3.8/bin/python -u tools/train.pyI am confused totally!!!

I set the environment variable export CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/general-user/frank/mlflow_ray/env/bin/python3 , then I clone a task and run it. This time no more reinstallation, the task use /general-user/frank/mlflow_ray/env/bin/python3 to run the experiment. But it stuck in the epoch10 when the first evaluation over and just stops there.

Not in docker mode. So I just need to set venvs_cache.path=/home/frank/env ? I do not think this works.

After I enqueue the experiment task into a queue, clearml agent always creates a new venv and installs pakcages.

And in the logs?

Hi TimelyMouse69 can you give any advice? Or can somebody else help? Thanks in advance.

Hi AgitatedDove14 can you give me some help?

I'm afraid what you're trying to do isn't a supported implementation.
You'll have to choose between using docker mode to have one virtual environment for everything or using the pip mode where you can used the cached virtual environments but you can't reuse the one you currently have.

Yes, I have tested it through another toy example and it reused the first run.

What I want is all the experiments use the same codes and the preinstalled python virtual env.

Could you test the following:
Without reusing the virtual environment you made manually:
Can you run a task twice and see if the second run is at least reusing the virtual environment of the first run?

So could you just setup your virtual environment with a task?

The logs outputs the 10 epoch's evaluation result and stops there. But it should run 50 epochs.

Hi TimelyMouse69 I am not going to cache the virtual env but reuse a preinstalled one. I have set agent.python_binary to use the virtual env python, but clearml use it to create another new virtual env not the preinstalled one. I have install the virtual env before I use clearml, so I want to reuse it.

To use a specific binary you can set in in the config: https://clear.ml/docs/latest/docs/configs/clearml_conf/#:~:text=python%20version%20(default)-,agent.python_binary,-(string)

But if you're trying to cache virtual environments you might be more interested in: https://clear.ml/docs/latest/docs/clearml_agent#environment-caching