Local changes are applied before installing requirements, right?

correct

Anyway sounds good! 🙂

LOL 🙂

first try the current setup using 

pip

, and if it fails, use 

poetry

 if 

poetry.lock

 exists

I guess the order here is not clear to me (the agent does the opposite), why would you start with pip if you are using poetry ?

Yeah I figured (2) would be the way to go actually 😄

Local changes are applied before installing requirements, right?

Tada! https://github.com/allegroai/clearml-agent/issues/90

Not necessarily on the same branch, no

Those are for specific packages, I'm wondering about the package managers as a whole

Here's how it failed for us 😅
poetry stores git related data in poetry.lock , so when you pip list , you get an internal package we have with its version, but no git reference, i.e. internal_module==1.2.3 instead of internal_module @ git+https://....@commit .
Then pip actually fails (our internal module is not on pypi), but poetry suceeds

clearml-agent  repo please 🙂

The other way will not work, as if you start with "pip" you cannot fail ... (if you fail it's in run time which is too late)

poetry

 stores git related data in ... you get an internal package we have with its version, but no git reference, i.e. 

internal_module==1.2.3

 instead of 

internal_module @H4dr1en

This seems like a bug with poetry (and I think I have run into this one), worth reporting it, no?

UnevenDolphin73 sounds great, any chance you can open a git issue on clearml-agent repo for this feature request ?

AgitatedDove14 for future reference this is indeed a PEP-610 related bug, fixed in https://python-poetry.org/blog/announcing-poetry-1.2.0a1/ . I see we can choose the pip version in the config, can we also set the poetry version used? Or is it updated from the lock file itself, or...?

Or some users that update their poetry.lock and some that update manually as they prefer to resolve on their own.

Hmmm maybe 

 I thought that was expected behavior from poetry side actually

I think this is the expected behavior, hence bug?!

Right so it uses whatever version is available on the agent.
Yeah it would be nice to have either a poetry_version (a-la https://github.com/allegroai/clearml-agent/blob/5afb604e3d53d3f09dd6de81fe0a494dacb2e94d/docs/clearml.conf#L62 ), rename the latter to manager_version , or just install from the captured environment, etc? 🤔

I'm with on this one 🙂 it better to make a company wide decision on these things and not allow too much flexibility (just two options to choose from, and it should be enough, I think)

The tl;dr is that some of our users like poetry and others prefer pip . Since pip install git+.... stores the git data, it seems trivial to first try and install based on pip , and only later on poetry , since the pip would crash with poetry as it stores git data elsewhere (in poetry.lock )

Task.force_requirements_env_freeze()

This might be very brittle, if users are running on a diff OS, or python versions...
I would actually go with:
you like poetry, update your lock file in git you do not use poetry, work on your own branch and delete poetry lock file
wdyt?

UnevenDolphin73 if the repo does not include a poetry file it will revert to pip

Haha, I've opened so many issues these past few days... Sure, np!

Maybe it's better to approach this the other way, if one uses Task.force_requirements_env_freeze() , then the locally updated packages aren't reflected in poetry 🤔

so you have a repo with poetry that some users update and some do not?
All working on the same branch ?

Should this be under the clearml or clearml-agent repo?

My current workaround is to use poetry and tell users to delete poetry.lock if they want their environment copied verbatim

Hmmm maybe 🤔 I thought that was expected behavior from poetry side actually

Fair enough 😄
Could be nice to be able to define the fallbacks under type maybe?
type: [ poetry, pip ] (current way under the hood) vs type: [ pip, poetry ]

for future reference this is indeed a PEP-610 related bug, f

👍

can we also set the 

poetry

 version used?....

Actually the agent assumes poetry is preinstalled (so whatever you already have on the docker) ...
That said, maybe we should install a specific version (after installing pip, we could do that if poetry is selected)
wdyt ?