Wait, should the clearml RC be installed outside the venv for the agent as well?
No need, the agent is totally independent
One way to circumvent this btw would be to also add/use the
--python
flag for
virtualenv
Notice that when creating the venv , the cmd that is used is basically pythonx.y -m virtualenv ...
By definition this will create a new venv based on the python that executes the venv.
With all that said, it might be there is a bug in virtualenv and in some cases, it does Not adhere to this restriction
Did you try using the latest agent RC? It's 1.2.0rc3
Same result 😞 This is frustrating, wtf happened :shocked_face_with_exploding_head:
This is also specifically the services queue worker I'm trying to debug 🤔
I also tried switching to dockerized mode now, getting the same issue 🤔
Any follow up thoughts SuccessfulKoala55 or CostlyOstrich36 ?
UnevenDolphin73 are you using the latest clearml RC?
I'll have yet another look at both the latest agent RC and at the docker-compose, thanks!
There was no "default" services agent btw, just the queue, I had to launch an agent myself (not sure if it's relevant)
Will try!
Curious - is there a temporary changelog for 1.2.0? 😁 Always fun to poke at the upcoming features
EDIT: Wait, should the clearml RC be installed outside the venv for the agent as well?
If it's the services queue worker, than it is running on the server. However, we didn't change anything in its configuration
So where should I install the latest clearml version? On the client that's running a task, or on the worker machine?
I've also followed https://clearml.slack.com/archives/CTK20V944/p1628333126247800 but it did not help
One way to circumvent this btw would be to also add/use the --python flag for virtualenv
Yes; I tried running it both outside venv and inside a venv. No idea why it uses 2.7?
Did you make sure the clearml-agent was not installed in a venv?
AgitatedDove14
I'll make a PR for it now, but the long story is that you have the full log, but the virtualenv version is not logged anywhere (the usual output from virtualenv just says which Python version is used, etc).
I also tried setting agent.python_binary: "/usr/bin/python3.8" but it still uses Python 2.7?
If it's the default services agent, you'll need to update the docker-compose for that
Still failing with 1.2.0rc3 😞 AgitatedDove14 any thoughts on your end?
https://github.com/allegroai/clearml-agent/pull/98 AgitatedDove14 😁
latest clearml is https://pypi.org/project/clearml/1.2.0rc0/
Hi UnevenDolphin73
Took a long time to figure out that there was a specific Python version with a specific virtualenv that was old ...
NICE!
Then the task requested to use Python 3.7, and that old virtualenv version was broken.
Yes, if the Task is using a specific python version it will first try to find this one (i.e. which python3.7 ) then use it to create the new venv
As a result -> Could the agent maybe also output the
virtualenv
version used with setting up the environment for the first time?
You mean into the Log ? (I think you have all the log from the get go, no?!)
EDIT:
I also tried setting
agent.python_binary: "/usr/bin/python3.8"
but it still uses Python 2.7?
I think that in docker mode this argument is ignored, as the assumption is it will break inside diff containers.
Okay this was a deep dive into clearml-agent code 😁
Took a long time to figure out that there was a specific Python version with a specific virtualenv that was old (Python 3.6.9 and Python 3.8 had latest virtualenv, but Python 3.7.5 had an old virtualenv).
Then the task requested to use Python 3.7, and that old virtualenv version was broken.
As a result -> Could the agent maybe also output the virtualenv version used with setting up the environment for the first time?
was consistent, whereas for some reason this old virtualenv decided to use python2.7 otherwise
Yes,
This sounds like a virtualenv bug I think it will not hurt to do both (obviously we have the information)
Thank you!!! 😍
I know, that should indeed be the default behaviour, but at least from my tests the use of --python ... was consistent, whereas for some reason this old virtualenv decided to use python2.7 otherwise 🤨
It's enough that the task requirements will contain this latest version
I'm using 1.1.6 (upgraded from 1.1.6rc0) - should I try 1.1.7rc0 or smth?
No changelog for now, we try to keep the commits clear and ordered 🙂