Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I'M Trying To Spin Up A Task On An Agent And Inside The Task I Have Two Packages That I'Ve Created Custom Versions Of And Specified A Git Repo For In The Requirements.Txt. Example With Hydra-Core And Omegaconf:

I'm trying to spin up a task on an agent and inside the task I have two packages that I've created custom versions of and specified a git repo for in the requirements.txt. Example with hydra-core and omegaconf:

# --------- core --------- #
darts==0.25.0
lightgbm  # for some reason these three packages needs to be installed after darts
prophet
catboost
# --------- hydra --------- #
# hydra-core and omegaconf are installed from a forked repo because the latest versions are not compatible with darts
# see: 

hydra-core @ 

omegaconf @ 

hydra-colorlog==1.2.0

However, during the install on the venv setup by the agent it seems to skip those packages:

Successfully installed pip-22.2.2
Collecting Cython
  Using cached Cython-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
Installing collected packages: Cython
Successfully installed Cython-3.0.2
Collecting darts==0.25.0
  Using cached darts-0.25.0-py3-none-any.whl (760 kB)
Collecting lightgbm
  Downloading lightgbm-4.1.0-py3-none-manylinux_2_28_x86_64.whl (3.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 15.3 MB/s eta 0:00:00
Collecting prophet
  Using cached prophet-1.1.4-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB)
Collecting catboost
  Downloading catboost-1.2.2-cp310-cp310-manylinux2014_x86_64.whl (98.7 MB)
2023-09-20 10:54:13
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.7/98.7 MB 12.2 MB/s eta 0:00:00
Collecting hydra-colorlog==1.2.0
  Using cached hydra_colorlog-1.2.0-py3-none-any.whl (3.6 kB)
Collecting matplotlib
  Downloading matplotlib-3.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/11.6 MB 16.7 MB/s eta 0:00:00

How would I include this in the install process?

  
  
Posted 5 months ago
Votes Newest

Answers 16


Actually this is not how it works, pip will install in any way it sees fit, and it is not consistent between versions (it has to do with dependency resolving)

Oh I see. What a pain. 🤣

You can configure the agent to first install specific packages, and only then others, just add the package names here:

That's an interesting solution. I'll keep that in mind as I work more with ClearML.

Thanks for your help Martin!

  
  
Posted 4 months ago

If I install using

pip install -r ./requirements.txt

then pip installs the packages in the order of the requirements file.

Actually this is not how it works, pip will install in any way it sees fit, and it is not consistent between versions (it has to do with dependency resolving)

However, during the installation process from ClearML, it installs the packages in order UNLESS there's a custom path provided, then it's saved for last

Correct because the custom (I'm assuming you mean install from git) package might need some of the wheels

The entire reason why I have the custom versions of the packages is I had to manually resolve a conflict between the darts package and hydra/omegaconf (hydra is working on an official solution but progress is slow).

You can configure the agent to first install specific packages, and only then others, just add the package names here:
None

I guess I could build a container.

always a good option

  
  
Posted 4 months ago

Sorry I disappeared (went on a well deserved vacation). The problem is happening because of the ordering of the install. If I install using pip install -r ./requirements.txt then pip installs the packages in the order of the requirements file. However, during the installation process from ClearML, it installs the packages in order UNLESS there's a custom path provided, then it's saved for last. The reason this breaks my code is I have later packages that depend on the custom packages, as such, they attempt to install the dependences from pypi. The entire reason why I have the custom versions of the packages is I had to manually resolve a conflict between the darts package and hydra/omegaconf (hydra is working on an official solution but progress is slow).

For now I can just exclude the hydra colorlog package but how would you suggest going about resolving this issue in a less hacky way? I guess I could build a container.

  
  
Posted 4 months ago

Hey @<1545216070686609408:profile|EnthusiasticCow4> , for requirements pointing to packages in git repositories you need to make sure that the environment the agent is running in has the valid credentials to access the repo. In your case ( git+ssh ) it means you need to have a pair of ssh keys, and the public key should be registered with the repo.

  
  
Posted 5 months ago

The git credentials are stored in the agent config and they work when I tested them on another project (not for setting up the environment but for downloading the repo of the task itself.)

  
  
Posted 5 months ago

If your git credentials are stored in the agent's clearml.conf it means these are a HTTPS username/password pair. But you specified that the package should be downloaded via git ssh, for which I assume you don't have credentials in agent's environment. So it can't authenticate with SSH, and PIP doesn't know how to switch from git+ssh to git+https, because the downloading of the package is done by PIP not by clearml.

And there probably are auth errors if you scroll through the entire log when pip will try to install your version of Hydra

  
  
Posted 5 months ago

Why? That's not how I authenticate. Also, if it was simply an issue with authentication wouldn't there be some error message in the log?

  
  
Posted 5 months ago

Then change from git+ssh to git+https

  
  
Posted 5 months ago

Unfortunately, that doesn't seem to have solved the problem. I tried the same thing with https and it seems to skip the lines with the @ symbol like it did before. Honestly, it seems more like it just isn't parsing those lines during the install.

Collecting darts==0.25.0
  Using cached darts-0.25.0-py3-none-any.whl (760 kB)
Collecting lightgbm
  Using cached lightgbm-4.1.0-py3-none-manylinux_2_28_x86_64.whl (3.1 MB)
Collecting prophet
  Using cached prophet-1.1.4-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB)
Collecting catboost
  Using cached catboost-1.2.2-cp310-cp310-manylinux2014_x86_64.whl (98.7 MB)
Collecting hydra-colorlog==1.2.0
  Using cached hydra_colorlog-1.2.0-py3-none-any.whl (3.6 kB)
Collecting matplotlib
  Using cached matplotlib-3.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB)
Collecting pyrootutils
  Using cached pyrootutils-1.0.4-py3-none-any.whl (5.8 kB)
Collecting pre-commit
  Using cached pre_commit-3.4.0-py2.py3-none-any.whl (203 kB)
Collecting rich
  Using cached rich-13.5.3-py3-none-any.whl (239 kB)

Here's the update requirements.txt:

# --------- core --------- #
darts==0.25.0
lightgbm  # for some reason these three packages needs to be installed after darts
prophet
catboost
# --------- hydra --------- #
# hydra-cre and omegaconf are installed from a forked repo because the latest versions are not compatible with darts
# see: 

hydra-core @ git+

omegaconf @ git+

hydra-colorlog==1.2.0
# --------- others --------- #
matplotlib             # plotting
pyrootutils            # standardizing the project root setup
pre-commit             # hooks for applying linters on commit
rich                   # beautiful text formatting in terminal
...

Also, the requirements file is being correctly displayed under " INSTALLED PACKAGES".

  
  
Posted 5 months ago

Oh, duh. I'll test that out. But I did have the agent.force_git_ssh_protocol: true

  
  
Posted 5 months ago

It's even attempting to install omegaconf but not from the repo, likely because it's a dependency of hydra-colorlog.

Collecting omegaconf<2.4,>=2.2
  Using cached omegaconf-2.2.3-py3-none-any.whl (79 kB)
  Using cached omegaconf-2.2.2-py3-none-any.whl (79 kB)
  Using cached omegaconf-2.2.1-py3-none-any.whl (78 kB)
  
  
Posted 5 months ago

@<1523701087100473344:profile|SuccessfulKoala55> You wouldn't happen to know what's going on here. :D

  
  
Posted 5 months ago

Can you paste here what inside "Installed package" to double check ?

  
  
Posted 5 months ago

It's verbatim from requirements as I pass that into ClearML.

  
  
Posted 5 months ago

@<1545216070686609408:profile|EnthusiasticCow4>
git+ssh:// will be converted automatically to git+https if you have user/pass ocnfigured in your clearml.conf on the agent machine.
More over, git packages are always installed After all other packages are installed (because pip cannot resolve the requirements inside the git repo in time)

  
  
Posted 5 months ago

Could you also provide the full log?

  
  
Posted 5 months ago
421 Views
16 Answers
5 months ago
4 months ago
Tags