Anyway, when I add the binary's path to PATH, it still won't work.
I call
task.set_base_docker(docker_image='my/docker/image', docker_setup_bash_script= ["export PATH=""/workspace/miniconda/bin:$PATH"])just after Task.init.
On execution, the agent installs all pip requirements with python binary at /usr/bin/python3.6
but eventually the task fails with, /workspace/miniconda/bin/python3: No module named clearml_agent

Hi CrookedWalrus33

docker_setup_bash_script= ["export PATH=""/workspace/miniconda/bin:$PATH"])

Oh I think you are correct, this should do the trick:
docker_setup_bash_script= ["export PATH=/workspace/miniconda/bin:$PATH", "export LOCAL_PYTHON=/workspace/miniconda/bin/python3"]This will make sure both agent and script execute on the same python

but to run a script inside a docker which already has the environment built in.

If this is already activated, the latest agent will just use the active venv ( docker_setup_bash_script=["conda activate /path"] )
for the above to work, you need the latest agent RC:
pip install clearml-agent==1.2.0rc7

The agent is using Bash (but when you add command line to the docker run, .bashrc is not executed, hence no conda in PATH)
Maybe add the full path to the conda executable:
ocker_setup_bash_script= [ "export PATH=""/workspace/miniconda/bin:$PATH", "export LOCAL_PYTHON=/workspace/miniconda/bin/python3","/workspace/miniconda/bin/conda activate /PATH_GOES_HERE"])

Hi CrookedWalrus33
the python version is auto detected and register in "manual execution" time (i.e. when you run your code on your machine).
That said this is a suggestion for the agent, and only if it can actually find the matching Python version it will use it, otherwise it will use whatever is
available (i.e. Look through the PATH environment for a matching pythonX.Y executable)
The easiest way to support would just make sure the python binary's path is added to the PATH env.
Does that make sense ?

It didn't work either. Still same error.
conda command was already in PATH and the conda activate is executed properly, but it prompts to run conda init (i.e conda wasn't configures at that shell):
` You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:

• bash
• fish
• tcsh
• xonsh
• zsh
• powershell
  See 'conda init --help' for more information and options.
  IMPORTANT: You may need to close and restart your shell after running 'conda init'.
  /workspace/miniconda/bin/python3: No module named clearml_agent `

CrookedWalrus33 can you send the entire log? (you can DM it to me)

AgitatedDove14
I'm not sure.
In my case I'm not trying to reproduce a local environment in the agent, but to run a script inside a docker which already has the environment built in.
The environment is conda based.

How can I specify the agent to use a specific conda environment inside the docker?

Hi CrookedWalrus33
By default it will pick the highest python in the PATH.
Then if you have a python version (in PATH) that matches the requested on on the Task, it will look for it.
Do you want to limit it to a specific python binary ?

AgitatedDove14
Yes, I'd like to point to a specific binary, which is in a conda environment.
(b.t.w how can I specify the python version on the Task?)

AgitatedDove14
It's still failing.
I updated clearml-agent to 1.2.0rc7 and also:
docker_setup_bash_script= [ "export PATH=""/workspace/miniconda/bin:$PATH", "export LOCAL_PYTHON=/workspace/miniconda/bin/python3", "conda activate"])But the conda activate (base env) returns:
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.

I noticed that conda is only configured to use with /bin/bash. Is it possible that the agent uses /bin/sh?

AgitatedDove14 .
Note that the actual error is /workspace/miniconda/bin/python3: No module named clearml_agent since all the packages (including clearml_agent) were already installed by the agent on the default (non conda) python binary.