Wait, should the clearml RC be installed outside the venv for the agent as well?

No need, the agent is totally independent

One way to circumvent this btw would be to also add/use the 

--python

 flag for 

virtualenv

Notice that when creating the venv , the cmd that is used is basically pythonx.y -m virtualenv ...
By definition this will create a new venv based on the python that executes the venv.
With all that said, it might be there is a bug in virtualenv and in some cases, it does Not adhere to this restriction

Did you try using the latest agent RC? It's 1.2.0rc3

Any follow up thoughts SuccessfulKoala55 or CostlyOstrich36 ?

Will try!

Curious - is there a temporary changelog for 1.2.0? 😁 Always fun to poke at the upcoming features

EDIT: Wait, should the clearml RC be installed outside the venv for the agent as well?

So where should I install the latest clearml version? On the client that's running a task, or on the worker machine?

I've also followed https://clearml.slack.com/archives/CTK20V944/p1628333126247800 but it did not help

Yes; I tried running it both outside venv and inside a venv. No idea why it uses 2.7?

AgitatedDove14
I'll make a PR for it now, but the long story is that you have the full log, but the virtualenv version is not logged anywhere (the usual output from virtualenv just says which Python version is used, etc).

I also tried setting agent.python_binary: "/usr/bin/python3.8" but it still uses Python 2.7?

If it's the default services agent, you'll need to update the docker-compose for that

Still failing with 1.2.0rc3 😞 AgitatedDove14 any thoughts on your end?

https://github.com/allegroai/clearml-agent/pull/98 AgitatedDove14 😁

latest clearml is https://pypi.org/project/clearml/1.2.0rc0/

I'm using 1.1.6 (upgraded from 1.1.6rc0) - should I try 1.1.7rc0 or smth?

No changelog for now, we try to keep the commits clear and ordered 🙂

Oh and clearml-agent==1.1.2

Hi UnevenDolphin73

Took a long time to figure out that there was a specific Python version with a specific virtualenv that was old ...

NICE!

Then the task requested to use Python 3.7, and that old virtualenv version was broken.

Yes, if the Task is using a specific python version it will first try to find this one (i.e. which python3.7 ) then use it to create the new venv

As a result -> Could the agent maybe also output the 

virtualenv

 version used with setting up the environment for the first time?

You mean into the Log ? (I think you have all the log from the get go, no?!)

EDIT:

I also tried setting 

agent.python_binary: "/usr/bin/python3.8"

 but it still uses Python 2.7?

I think that in docker mode this argument is ignored, as the assumption is it will break inside diff containers.

It's enough that the task requirements will contain this latest version

UnevenDolphin73 are you using the latest clearml RC?

I also tried switching to dockerized mode now, getting the same issue 🤔

I'll have yet another look at both the latest agent RC and at the docker-compose, thanks!

There was no "default" services agent btw, just the queue, I had to launch an agent myself (not sure if it's relevant)

was consistent, whereas for some reason this old virtualenv decided to use python2.7 otherwise 

Yes,
This sounds like a virtualenv bug I think it will not hurt to do both (obviously we have the information)

Thank you!!! 😍

If it's the services queue worker, than it is running on the server. However, we didn't change anything in its configuration

I know, that should indeed be the default behaviour, but at least from my tests the use of --python ... was consistent, whereas for some reason this old virtualenv decided to use python2.7 otherwise 🤨

Okay this was a deep dive into clearml-agent code 😁
Took a long time to figure out that there was a specific Python version with a specific virtualenv that was old (Python 3.6.9 and Python 3.8 had latest virtualenv, but Python 3.7.5 had an old virtualenv).
Then the task requested to use Python 3.7, and that old virtualenv version was broken.

As a result -> Could the agent maybe also output the virtualenv version used with setting up the environment for the first time?

Did you make sure the clearml-agent was not installed in a venv?

Yeah

One way to circumvent this btw would be to also add/use the --python flag for virtualenv

Same result 😞 This is frustrating, wtf happened :shocked_face_with_exploding_head:
This is also specifically the services queue worker I'm trying to debug 🤔