CostlyOstrich36 ok the issue is not the clearml-agent version, it is the conda environment I'm trying to run with. I usually run my agents from one conda environment and when I pass CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1, it just uses the conda interpreter and all is good. When I try to do this with a different conda env, this does not work and tries forcibly to create a new venv. any idea why?

Edit: if it is any help, the conda env that works is installed in the conda install dir, and the not-working conda env is installed on a different local disk.

Digging through the log, this is the main difference:
This is an example of a good detection of current conda env:
Interpreter 'python3.8' found at '/home/USERNAME/miniconda3/envs/torch/bin/python3.8'

and this is a bad example:
Interpreter 'python3.8' found at '/usr/bin/python3.8'

This is bad since the python interpreter in the second conda env is python3.9 and not 3.8. This is despite the fact that agent.default_python=3.9 which is the correct version.