Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everybody, I'M Running Experiments Inside A Docker Which Includes Multiple Python Instances, Some Of Them Are Inside Conda Environments. How Can I Specify The Agent To Use A Specific Conda Environment Inside The Docker?

Hi everybody,
I'm running experiments inside a docker which includes multiple python instances, some of them are inside conda environments.
How can I specify the agent to use a specific conda environment inside the docker?

  
  
Posted one year ago
Votes Newest

Answers 11


AgitatedDove14
I'm not sure.
In my case I'm not trying to reproduce a local environment in the agent, but to run a script inside a docker which already has the environment built in.
The environment is conda based.

  
  
Posted one year ago

Hi CrookedWalrus33
the python version is auto detected and register in "manual execution" time (i.e. when you run your code on your machine).
That said this is a suggestion for the agent, and only if it can actually find the matching Python version it will use it, otherwise it will use whatever is
available (i.e. Look through the PATH environment for a matching pythonX.Y executable)
The easiest way to support would just make sure the python binary's path is added to the PATH env.
Does that make sense ?

  
  
Posted one year ago

AgitatedDove14
Yes, I'd like to point to a specific binary, which is in a conda environment.
(b.t.w how can I specify the python version on the Task?)

  
  
Posted one year ago

CrookedWalrus33 can you send the entire log? (you can DM it to me)

  
  
Posted one year ago

It didn't work either. Still same error.
conda command was already in PATH and the conda activate is executed properly, but it prompts to run conda init (i.e conda wasn't configures at that shell):
` You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:

  • bash
  • fish
  • tcsh
  • xonsh
  • zsh
  • powershell
    See 'conda init --help' for more information and options.
    IMPORTANT: You may need to close and restart your shell after running 'conda init'.
    /workspace/miniconda/bin/python3: No module named clearml_agent `
  
  
Posted one year ago

How can I specify the agent to use a specific conda environment inside the docker?

Hi CrookedWalrus33
By default it will pick the highest python in the PATH.
Then if you have a python version (in PATH) that matches the requested on on the Task, it will look for it.
Do you want to limit it to a specific python binary ?

  
  
Posted one year ago

Anyway, when I add the binary's path to PATH, it still won't work.
I call
task.set_base_docker(docker_image='my/docker/image', docker_setup_bash_script= ["export PATH=""/workspace/miniconda/bin:$PATH"])just after Task.init.
On execution, the agent installs all pip requirements with python binary at /usr/bin/python3.6
but eventually the task fails with, /workspace/miniconda/bin/python3: No module named clearml_agent

  
  
Posted one year ago

Hi CrookedWalrus33

docker_setup_bash_script= ["export PATH=""/workspace/miniconda/bin:$PATH"])

Oh I think you are correct, this should do the trick:
docker_setup_bash_script= ["export PATH=/workspace/miniconda/bin:$PATH", "export LOCAL_PYTHON=/workspace/miniconda/bin/python3"]This will make sure both agent and script execute on the same python

but to run a script inside a docker which already has the environment built in.

If this is already activated, the latest agent will just use the active venv ( docker_setup_bash_script=["conda activate /path"] )
for the above to work, you need the latest agent RC:
pip install clearml-agent==1.2.0rc7

  
  
Posted one year ago

AgitatedDove14
It's still failing.
I updated clearml-agent to 1.2.0rc7 and also:
docker_setup_bash_script= [ "export PATH=""/workspace/miniconda/bin:$PATH", "export LOCAL_PYTHON=/workspace/miniconda/bin/python3", "conda activate"])But the conda activate (base env) returns:
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.

I noticed that conda is only configured to use with /bin/bash. Is it possible that the agent uses /bin/sh?

  
  
Posted one year ago

The agent is using Bash (but when you add command line to the docker run, .bashrc is not executed, hence no conda in PATH)
Maybe add the full path to the conda executable:
ocker_setup_bash_script= [ "export PATH=""/workspace/miniconda/bin:$PATH", "export LOCAL_PYTHON=/workspace/miniconda/bin/python3","/workspace/miniconda/bin/conda activate /PATH_GOES_HERE"])

  
  
Posted one year ago

AgitatedDove14 .
Note that the actual error is /workspace/miniconda/bin/python3: No module named clearml_agent since all the packages (including clearml_agent) were already installed by the agent on the default (non conda) python binary.

  
  
Posted one year ago