Sorry I disappeared (went on a well deserved vacation). The problem is happening because of the ordering of the install. If I install using pip install -r ./requirements.txt then pip installs the packages in the order of the requirements file. However, during the installation process from ClearML, it installs the packages in order UNLESS there's a custom path provided, then it's saved for last. The reason this breaks my code is I have later packages that depend on the custom packages, as such, they attempt to install the dependences from pypi. The entire reason why I have the custom versions of the packages is I had to manually resolve a conflict between the darts package and hydra/omegaconf (hydra is working on an official solution but progress is slow).
For now I can just exclude the hydra colorlog package but how would you suggest going about resolving this issue in a less hacky way? I guess I could build a container.
Actually this is not how it works, pip will install in any way it sees fit, and it is not consistent between versions (it has to do with dependency resolving)
Oh I see. What a pain. 🤣
You can configure the agent to first install specific packages, and only then others, just add the package names here:
That's an interesting solution. I'll keep that in mind as I work more with ClearML.
Thanks for your help Martin!
@<1523701087100473344:profile|SuccessfulKoala55> You wouldn't happen to know what's going on here. :D
If your git credentials are stored in the agent's clearml.conf it means these are a HTTPS username/password pair. But you specified that the package should be downloaded via git ssh, for which I assume you don't have credentials in agent's environment. So it can't authenticate with SSH, and PIP doesn't know how to switch from git+ssh to git+https, because the downloading of the package is done by PIP not by clearml.
And there probably are auth errors if you scroll through the entire log when pip will try to install your version of Hydra
Hey @<1545216070686609408:profile|EnthusiasticCow4> , for requirements pointing to packages in git repositories you need to make sure that the environment the agent is running in has the valid credentials to access the repo. In your case ( git+ssh ) it means you need to have a pair of ssh keys, and the public key should be registered with the repo.
@<1545216070686609408:profile|EnthusiasticCow4>git+ssh:// will be converted automatically to git+https if you have user/pass ocnfigured in your clearml.conf on the agent machine.
More over, git packages are always installed After all other packages are installed (because pip cannot resolve the requirements inside the git repo in time)
It's verbatim from requirements as I pass that into ClearML.
The git credentials are stored in the agent config and they work when I tested them on another project (not for setting up the environment but for downloading the repo of the task itself.)
Oh, duh. I'll test that out. But I did have the agent.force_git_ssh_protocol: true
Can you paste here what inside "Installed package" to double check ?
Unfortunately, that doesn't seem to have solved the problem. I tried the same thing with https and it seems to skip the lines with the @ symbol like it did before. Honestly, it seems more like it just isn't parsing those lines during the install.
Collecting darts==0.25.0
Using cached darts-0.25.0-py3-none-any.whl (760 kB)
Collecting lightgbm
Using cached lightgbm-4.1.0-py3-none-manylinux_2_28_x86_64.whl (3.1 MB)
Collecting prophet
Using cached prophet-1.1.4-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB)
Collecting catboost
Using cached catboost-1.2.2-cp310-cp310-manylinux2014_x86_64.whl (98.7 MB)
Collecting hydra-colorlog==1.2.0
Using cached hydra_colorlog-1.2.0-py3-none-any.whl (3.6 kB)
Collecting matplotlib
Using cached matplotlib-3.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB)
Collecting pyrootutils
Using cached pyrootutils-1.0.4-py3-none-any.whl (5.8 kB)
Collecting pre-commit
Using cached pre_commit-3.4.0-py2.py3-none-any.whl (203 kB)
Collecting rich
Using cached rich-13.5.3-py3-none-any.whl (239 kB)
Here's the update requirements.txt:
# --------- core --------- #
darts==0.25.0
lightgbm # for some reason these three packages needs to be installed after darts
prophet
catboost
# --------- hydra --------- #
# hydra-cre and omegaconf are installed from a forked repo because the latest versions are not compatible with darts
# see:
hydra-core @ git+
omegaconf @ git+
hydra-colorlog==1.2.0
# --------- others --------- #
matplotlib # plotting
pyrootutils # standardizing the project root setup
pre-commit # hooks for applying linters on commit
rich # beautiful text formatting in terminal
...
Also, the requirements file is being correctly displayed under " INSTALLED PACKAGES".
If I install using
pip install -r ./requirements.txt
then pip installs the packages in the order of the requirements file.
Actually this is not how it works, pip will install in any way it sees fit, and it is not consistent between versions (it has to do with dependency resolving)
However, during the installation process from ClearML, it installs the packages in order UNLESS there's a custom path provided, then it's saved for last
Correct because the custom (I'm assuming you mean install from git) package might need some of the wheels
The entire reason why I have the custom versions of the packages is I had to manually resolve a conflict between the darts package and hydra/omegaconf (hydra is working on an official solution but progress is slow).
You can configure the agent to first install specific packages, and only then others, just add the package names here:
None
I guess I could build a container.
always a good option
It's even attempting to install omegaconf but not from the repo, likely because it's a dependency of hydra-colorlog.
Collecting omegaconf<2.4,>=2.2
Using cached omegaconf-2.2.3-py3-none-any.whl (79 kB)
Using cached omegaconf-2.2.2-py3-none-any.whl (79 kB)
Using cached omegaconf-2.2.1-py3-none-any.whl (78 kB)
Why? That's not how I authenticate. Also, if it was simply an issue with authentication wouldn't there be some error message in the log?