CostlyOstrich36 it worked! thank you so much :grinning_face_with_star_eyes:

HugeArcticwolf77 , what if you specify the exact python binary from conda using CLEARML_AGENT_SKIP_PIP_VENV_INSTALL?

CostlyOstrich36 ok the issue is not the clearml-agent version, it is the conda environment I'm trying to run with. I usually run my agents from one conda environment and when I pass CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1, it just uses the conda interpreter and all is good. When I try to do this with a different conda env, this does not work and tries forcibly to create a new venv. any idea why?

Edit: if it is any help, the conda env that works is installed in the conda install dir, and the not-working conda env is installed on a different local disk.

Digging through the log, this is the main difference:
This is an example of a good detection of current conda env:
Interpreter 'python3.8' found at '/home/USERNAME/miniconda3/envs/torch/bin/python3.8'

and this is a bad example:
Interpreter 'python3.8' found at '/usr/bin/python3.8'

This is bad since the python interpreter in the second conda env is python3.9 and not 3.8. This is despite the fact that agent.default_python=3.9 which is the correct version.

Hi HugeArcticwolf77 , did you spin new agent versions? What did you have before and what do you have now? Can you check if you revert to the previous version it works out?