Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Have Install A Python Environment By Virtualenv Tool, Let'S Say

I have install a python environment by virtualenv tool, let's say /home/frank/env and python is /home/frank/env/bin/python3. How to reuse the virtualenv by setting clearml agent?

  
  
Posted one year ago
Votes Newest

Answers 18


Could you test the following:
Without reusing the virtual environment you made manually:
Can you run a task twice and see if the second run is at least reusing the virtual environment of the first run?

  
  
Posted one year ago

Yes, I have tested it through another toy example and it reused the first run.

  
  
Posted one year ago

Not in docker mode. So I just need to set venvs_cache.path=/home/frank/env ? I do not think this works.

  
  
Posted one year ago

I'm afraid what you're trying to do isn't a supported implementation.
You'll have to choose between using docker mode to have one virtual environment for everything or using the pip mode where you can used the cached virtual environments but you can't reuse the one you currently have.

  
  
Posted one year ago

Hi TimelyMouse69 I am not going to cache the virtual env but reuse a preinstalled one. I have set agent.python_binary to use the virtual env python, but clearml use it to create another new virtual env not the preinstalled one. I have install the virtual env before I use clearml, so I want to reuse it.

  
  
Posted one year ago

What I want is all the experiments use the same codes and the preinstalled python virtual env.

  
  
Posted one year ago

After I enqueue the experiment task into a queue, clearml agent always creates a new venv and installs pakcages.

  
  
Posted one year ago

And in the logs?

  
  
Posted one year ago

Hi AgitatedDove14 can you give me some help?

  
  
Posted one year ago

Of course I did.

  
  
Posted one year ago

So could you just setup your virtual environment with a task?

  
  
Posted one year ago

Hi TimelyMouse69 can you give any advice? Or can somebody else help? Thanks in advance.

  
  
Posted one year ago

The logs outputs the 10 epoch's evaluation result and stops there. But it should run 50 epochs.

  
  
Posted one year ago

I set the environment variable export CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/general-user/frank/mlflow_ray/env/bin/python3 , then I clone a task and run it. This time no more reinstallation, the task use /general-user/frank/mlflow_ray/env/bin/python3 to run the experiment. But it stuck in the epoch10 when the first evaluation over and just stops there.

  
  
Posted one year ago

Even I only set agent.venvs_cache.path=/home/frank/env in the clearml.conf, the train still failed to start. ClearML always create a new venvs and install Cython and some other packages, so weird!
... Installing collected packages: Cython Successfully installed Cython-0.29.32 Adding venv into cache: /general-user/frank/.clearml/venvs-builds/3.8 Running task id [42a050853c43445ebf9248bd4aa54091]: [.]$ /general-user/frank/.clearml/venvs-builds/3.8/bin/python -u tools/train.pyI am confused totally!!!

  
  
Posted one year ago

To use a specific binary you can set in in the config: https://clear.ml/docs/latest/docs/configs/clearml_conf/#:~:text=python%20version%20(default)-,agent.python_binary,-(string)

But if you're trying to cache virtual environments you might be more interested in: https://clear.ml/docs/latest/docs/clearml_agent#environment-caching

  
  
Posted one year ago

ThoughtfulBadger56 Have you uncommented the existing venvs_cache section in the config file?
https://clear.ml/docs/latest/docs/clearml_agent#virtual-environment-reuse

  
  
Posted one year ago

I have install a python environment by virtualenv tool, let's say

/home/frank/env

and python is

/home/frank/env/bin/python3.

How to reuse the virtualenv by setting clearml agent?

So the agent is already caching the entire venv for you, nothing to worry about, just make sure you have this line in clearml:
https://github.com/allegroai/clearml-agent/blob/249b51a31bee97d63f41c6d5542e657962008b68/docs/clearml.conf#L131
No need to provide it an existing venv, if you are running in docker mode, you can provide
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/home/frank/env/bin/python3

  
  
Posted one year ago
614 Views
18 Answers
one year ago
one year ago
Tags