This is why I'm wondering if running the initial experiment in local in a venv is the reason why clearml is struggling to reproduce the experiment
Hi @<1546303254386708480:profile|DisgustedBear75> , what do you mean?
You get different results or your experiment fails?
Running in venv mode can be more prone to failure if you're running between different operating systems & python versions.
The default behavior of ClearML when running locally is to detect the packages used in the code execution (You can also provide specific packages manually or override auto detection entirely) and log them in the backend.
When a worker in a virtual environment mode picks up an experiment it will try to create a virtual environment using local python.
If some specific wheel versions aren't available to the python version the worker is running inside the virtual environment, the experiment will fail.
Is this your scenario?
The experiment fails.
Yes this is my scenario.
Basically I'm using this command "Task.force_requirements_env_freeze(requirements_file='requirements.txt')" on a requirement file that I know is working in local (if i set up a venv with that requirement file, the script is running)
But when I clone and rerun the experiment, clearml isnt able to install the requirements (i checked and the same version of python is used)