VivaciousWalrus99
Yes this is odd:
1608392232071 spectralab:gpu0 DEBUG New python executable in /cs/usr/gal.hyams/.trains/venvs-builds/3.7/bin/python2So it thinks it has python v3.7 but it is using python2 in the venv...
In your trains.conf file, set agent.python_binary to the python3.7 binary. It should be something like:
agent.python_binary=/path/to/python/python3.7

VivaciousWalrus99 any chance the original Task was executed with python2 ?
what do you have for:
ls -la /cs/usr/gal.hyams/.trains/venvs-builds/3.7/bin/

Thanks for the quick response, Martin! The original experiment was executed with python3.7

still getting /cs/usr/gal.hyams/.trains/venvs-builds/3.7/bin/python2

Hi VivaciousWalrus99
Could you attach the log of the run ?
By default it will use the python it is running with.
Any chance the original experiment was executed with python2 ?

AgitatedDove14

The original was executed with python3. ls -la /cs/usr/gal.hyams/.trains/venvs-builds/3.7/bin/ Contines python2

New python executable in /cs/labs/gavish/gal.hyams/.clearml/venvs-builds/3.7/bin/python2 Also creating executable in /cs/labs/gavish/gal.hyams/.clearml/venvs-builds/3.7/bin/python Installing setuptools, pkg_resources, pip, wheel... 2021-01-11 14:46:23 done.Running virtualenv with interpreter /usr/bin/python2

Thanks Martin. It did not work. I've tried changing my user the default python and pip to 3 as well: ln -s which python3 ~/bin/python..