Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Is It Possible To Give The Agent Access To Install Private Pip Packages (Needs To Be Installed From The Repo)?

Is it possible to give the agent access to install private pip packages (needs to be installed from the repo)?
ERROR: Could not find a version that satisfies the requirement private_package ERROR: No matching distribution found for private_package RequirementsManager handler <clearml_agent.helper.package.external_req.ExternalRequirements object at 0x7faccd9e16d8> raised exception: Failed installing GIT/HTTPs package 'private_package' clearml_agent: ERROR: Could not install task requirements! Failed installing GIT/HTTPs package 'private_package'

  
  
Posted 2 years ago
Votes Newest

Answers 30


p.s. you should remove this line 🙂
extra_index_url: ["git@github.com:salimmj/xxxx"]

  
  
Posted 2 years ago

... more-itertools==8.6.0 -e git+git@github.com:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package msgpack==1.0.2 msgpack-numpy==0.4.7.1 ...

  
  
Posted 2 years ago

Well this doesn’t work
pip install -e

  
  
Posted 2 years ago

Try to manually edit the "Installed Packages" (right click the Task, select "reset", now you can edit the section)
and change it to :
-e git+ssh@github.com:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package(assuming " pip install -e mailto:git+ssh@github.com :user/... " will work, should solve the issue )

  
  
Posted 2 years ago

I think it works, I’m fixing something else that came up.

  
  
Posted 2 years ago

If venv works inside containers that’s even better. We actually have custom containers that build on master merges. I wonder if using our own containers which should have most the deps will work better than a simpler container.

  
  
Posted 2 years ago

Our code is tightly integrated with protobuffers which needs to be re-compiled every now and then. We have a script to do that. If that’s not done, some imports end up failing.

  
  
Posted 2 years ago

I hadn’t enabled that line when the failure happened.

  
  
Posted 2 years ago

Can you copy the "Installed Packages" here, and point to the package causing the issue?

  
  
Posted 2 years ago

Unfortunately not yet in venv mode. What would you have put there?

  
  
Posted 2 years ago

I am already forcing ssh auth

  
  
Posted 2 years ago

The commit is valid for sure.

  
  
Posted 2 years ago

AgitatedDove14 can I specify a script to be run after pip install packages is done? I see that it’s possible in docker mode.

  
  
Posted 2 years ago

If you were to add this, where would you put it? I can use a modified version of clearml-agent

  
  
Posted 2 years ago

AgitatedDove14 this works: pip install git+ssh://git@github.com/user/repo.git

  
  
Posted 2 years ago

Nice!

  
  
Posted 2 years ago

It is indeed autopopulated by init

  
  
Posted 2 years ago

I wonder if using our own containers which should have most the deps will work better than a simpler container.

Why not, it's transparent, just run in --docker mode and provide a default docker image if the Task doesn't specify one.

  
  
Posted 2 years ago

Great find! So a pip upgrade should fix it hopefully.

  
  
Posted 2 years ago

I’m not sure if 

https

 will work because I want to use ssh keys for creds.

BTW: I was not aware github provide pypi like artifactory, do they ?
Regrading SSH keys, they are passed from the host machine (i.e. in venv mode it will use the SSH keys from the user running the agent, and n docker mode, they are automatically mapped into the container)

  
  
Posted 2 years ago

LazyTurkey38 I think this is caused by new versions of pip to report the wrong link:
https://github.com/bwoodsend/pip/commit/f533671b0ca9689855b7bdda67f44108387fe2a9

  
  
Posted 2 years ago

-e

:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package

Is this the correct link to the repo and a valid commit id ?

Can you post a few more lines from the agent's log ?
Something is failing to install I'm just not sure what

  
  
Posted 2 years ago

Is there a way to make it use ssh+git instead of git+git ? Maybe add a force_ssh_pip_install to the agent config?

  
  
Posted 2 years ago

First let's verify with the manual change, but yes

  
  
Posted 2 years ago

The agent is installing the "Installed Paclages" section of the Task (think of it as requirements.txt)
And again, what do you have there? Is it the outcome of the Task.init auto populating it?

  
  
Posted 2 years ago

The private_package can be installed by doing pip install git+ssh://git@github.com/user/private_package.git but the agent is trying to do pip install private_package which won’t work.

  
  
Posted 2 years ago

I see..
Generally speaking If that is the case, I would think it might be better to use the docker mode, it offers way more stable environment, regardless on the host machine runinng the agent. Notice there is no need to use custom containers, as the agent will basically run the venv process, only inside a container, allowing you to reuse offf the shelf containers.

If you were to add this, where would you put it? I can use a modified version of 

clearml-agent

Yep, that would be the place, basically you can add it here,
https://github.com/allegroai/clearml-agent/blob/bd411a19843fbb1e063b131e830a4515233bdf04/clearml_agent/commands/worker.py#L2095
You can get the actual script from the Task object current_task .
Please feel free to PR once you get it working, I'll be more than happy to help

  
  
Posted 2 years ago

Ok!

  
  
Posted 2 years ago

This means that in your "Installed packages" you should see the line:
Notice that this is not a pypi artifactory (i.e. a server to add to the extra index url for pip), this is a direct pip install from a git repository, hence it should be listed in the "installed packages".
If this is the way the package was installed locally, you should have had this line in the installed packages.
The clearml agent should take care of the authentication for you (specifically here, it should do nothing).
If you need all git references converted to SSH authentication (by default it is https with user/pass), set
force_git_ssh_protocol: true in your agent's clearml.conf file
https://github.com/allegroai/clearml-agent/blob/bd411a19843fbb1e063b131e830a4515233bdf04/docs/clearml.conf#L25

  
  
Posted 2 years ago

It doesn’t install it automatically, I think I need to specify it somewhere, see the above error. Or am I misunderstanding?

  
  
Posted 2 years ago
537 Views
30 Answers
2 years ago
one year ago
Tags