The private_package can be installed by doing pip install git+ssh://git@github.com/user/private_package.git but the agent is trying to do pip install private_package which won’t work.

Ok!

Try to manually edit the "Installed Packages" (right click the Task, select "reset", now you can edit the section)
and change it to :
-e git+ssh@github.com:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package(assuming " pip install -e mailto:git+ssh@github.com :user/... " will work, should solve the issue )

-e

:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package

Is this the correct link to the repo and a valid commit id ?

Can you post a few more lines from the agent's log ?
Something is failing to install I'm just not sure what

AgitatedDove14 can I specify a script to be run after pip install packages is done? I see that it’s possible in docker mode.

Well this doesn’t work
pip install -e

I see..
Generally speaking If that is the case, I would think it might be better to use the docker mode, it offers way more stable environment, regardless on the host machine runinng the agent. Notice there is no need to use custom containers, as the agent will basically run the venv process, only inside a container, allowing you to reuse offf the shelf containers.

If you were to add this, where would you put it? I can use a modified version of 

clearml-agent

Yep, that would be the place, basically you can add it here,
https://github.com/allegroai/clearml-agent/blob/bd411a19843fbb1e063b131e830a4515233bdf04/clearml_agent/commands/worker.py#L2095
You can get the actual script from the Task object current_task .
Please feel free to PR once you get it working, I'll be more than happy to help

Is there a way to make it use ssh+git instead of git+git ? Maybe add a force_ssh_pip_install to the agent config?

LazyTurkey38 I think this is caused by new versions of pip to report the wrong link:
https://github.com/bwoodsend/pip/commit/f533671b0ca9689855b7bdda67f44108387fe2a9

Can you copy the "Installed Packages" here, and point to the package causing the issue?

I hadn’t enabled that line when the failure happened.

First let's verify with the manual change, but yes

The commit is valid for sure.

I’m not sure if 

https

 will work because I want to use ssh keys for creds.

BTW: I was not aware github provide pypi like artifactory, do they ?
Regrading SSH keys, they are passed from the host machine (i.e. in venv mode it will use the SSH keys from the user running the agent, and n docker mode, they are automatically mapped into the container)

If you were to add this, where would you put it? I can use a modified version of clearml-agent

p.s. you should remove this line 🙂
extra_index_url: ["git@github.com:salimmj/xxxx"]

Great find! So a pip upgrade should fix it hopefully.

Nice!

The agent is installing the "Installed Paclages" section of the Task (think of it as requirements.txt)
And again, what do you have there? Is it the outcome of the Task.init auto populating it?

AgitatedDove14 this works: pip install git+ssh://git@github.com/user/repo.git

Unfortunately not yet in venv mode. What would you have put there?

It is indeed autopopulated by init

This means that in your "Installed packages" you should see the line:
Notice that this is not a pypi artifactory (i.e. a server to add to the extra index url for pip), this is a direct pip install from a git repository, hence it should be listed in the "installed packages".
If this is the way the package was installed locally, you should have had this line in the installed packages.
The clearml agent should take care of the authentication for you (specifically here, it should do nothing).
If you need all git references converted to SSH authentication (by default it is https with user/pass), set
force_git_ssh_protocol: true in your agent's clearml.conf file
https://github.com/allegroai/clearml-agent/blob/bd411a19843fbb1e063b131e830a4515233bdf04/docs/clearml.conf#L25

If venv works inside containers that’s even better. We actually have custom containers that build on master merges. I wonder if using our own containers which should have most the deps will work better than a simpler container.

I wonder if using our own containers which should have most the deps will work better than a simpler container.

Why not, it's transparent, just run in --docker mode and provide a default docker image if the Task doesn't specify one.

It doesn’t install it automatically, I think I need to specify it somewhere, see the above error. Or am I misunderstanding?

I am already forcing ssh auth

... more-itertools==8.6.0 -e git+git@github.com:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package msgpack==1.0.2 msgpack-numpy==0.4.7.1 ...

I think it works, I’m fixing something else that came up.

Our code is tightly integrated with protobuffers which needs to be re-compiled every now and then. We have a script to do that. If that’s not done, some imports end up failing.