CostlyOstrich36 ok the issue is not the clearml-agent version, it is the conda environment I'm trying to run with. I usually run my agents from one conda environment and when I pass CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1, it just uses the conda interpreter and all is good. When I try to do this with a different conda env, this does not work and tries forcibly to create a new venv. any idea why?

Edit: if it is any help, the conda env that works is installed in the conda install dir, and the not-working conda env is installed on a different local disk.

Digging through the log, this is the main difference:
This is an example of a good detection of current conda env:
Interpreter 'python3.8' found at '/home/USERNAME/miniconda3/envs/torch/bin/python3.8'

and this is a bad example:
Interpreter 'python3.8' found at '/usr/bin/python3.8'

This is bad since the python interpreter in the second conda env is python3.9 and not 3.8. This is despite the fact that agent.default_python=3.9 which is the correct version.

Hi HugeArcticwolf77 , did you spin new agent versions? What did you have before and what do you have now? Can you check if you revert to the previous version it works out?

CostlyOstrich36 it worked! thank you so much :grinning_face_with_star_eyes:

HugeArcticwolf77 , what if you specify the exact python binary from conda using CLEARML_AGENT_SKIP_PIP_VENV_INSTALL?