so you have a repo with poetry that some users update and some do not?
All working on the same branch ?
UnevenDolphin73 if the repo does not include a poetry file it will revert to pip
Should this be under the clearml
or clearml-agent
repo?
I'm with on this one 🙂 it better to make a company wide decision on these things and not allow too much flexibility (just two options to choose from, and it should be enough, I think)
Maybe it's better to approach this the other way, if one uses Task.force_requirements_env_freeze()
, then the locally updated packages aren't reflected in poetry
🤔
Right so it uses whatever version is available on the agent.
Yeah it would be nice to have either a poetry_version
(a-la https://github.com/allegroai/clearml-agent/blob/5afb604e3d53d3f09dd6de81fe0a494dacb2e94d/docs/clearml.conf#L62 ), rename the latter to manager_version
, or just install from the captured environment, etc? 🤔
Haha, I've opened so many issues these past few days... Sure, np!
Hmmm maybe
I thought that was expected behavior from poetry side actually
I think this is the expected behavior, hence bug?!
poetry
stores git related data in ... you get an internal package we have with its version, but no git reference, i.e.
internal_module==1.2.3
instead of
internal_module @H4dr1en
This seems like a bug with poetry (and I think I have run into this one), worth reporting it, no?
first try the current setup using
pip
, and if it fails, use
poetry
if
poetry.lock
exists
I guess the order here is not clear to me (the agent does the opposite), why would you start with pip if you are using poetry ?
Local changes are applied before installing requirements, right?
The other way will not work, as if you start with "pip" you cannot fail ... (if you fail it's in run time which is too late)
AgitatedDove14 for future reference this is indeed a PEP-610 related bug, fixed in https://python-poetry.org/blog/announcing-poetry-1.2.0a1/ . I see we can choose the pip
version in the config, can we also set the poetry
version used? Or is it updated from the lock file itself, or...?
UnevenDolphin73 sounds great, any chance you can open a git issue on clearml-agent repo for this feature request ?
Fair enough 😄
Could be nice to be able to define the fallbacks under type
maybe?type: [ poetry, pip ]
(current way under the hood) vs type: [ pip, poetry ]
Task.force_requirements_env_freeze()
This might be very brittle, if users are running on a diff OS, or python versions...
I would actually go with:
you like poetry, update your lock file in git you do not use poetry, work on your own branch and delete poetry lock file
wdyt?
My current workaround is to use poetry
and tell users to delete poetry.lock
if they want their environment copied verbatim
Here's how it failed for us 😅poetry
stores git related data in poetry.lock
, so when you pip list
, you get an internal package we have with its version, but no git reference, i.e. internal_module==1.2.3
instead of internal_module @ git+https://....@commit
.
Then pip
actually fails (our internal module is not on pypi), but poetry
suceeds
Those are for specific packages, I'm wondering about the package managers as a whole
Yeah I figured (2) would be the way to go actually 😄
The tl;dr is that some of our users like poetry
and others prefer pip
. Since pip install git+....
stores the git data, it seems trivial to first try and install based on pip
, and only later on poetry
, since the pip
would crash with poetry
as it stores git data elsewhere (in poetry.lock
)
for future reference this is indeed a PEP-610 related bug, f
👍
can we also set the
poetry
version used?....
Actually the agent assumes poetry is preinstalled (so whatever you already have on the docker) ...
That said, maybe we should install a specific version (after installing pip, we could do that if poetry is selected)
wdyt ?
Hmmm maybe 🤔 I thought that was expected behavior from poetry side actually
Or some users that update their poetry.lock
and some that update manually as they prefer to resolve on their own.
Local changes are applied before installing requirements, right?
correct