I wonder if using our own containers which should have most the deps will work better than a simpler container.

Why not, it's transparent, just run in --docker mode and provide a default docker image if the Task doesn't specify one.

If venv works inside containers that’s even better. We actually have custom containers that build on master merges. I wonder if using our own containers which should have most the deps will work better than a simpler container.

I see..
Generally speaking If that is the case, I would think it might be better to use the docker mode, it offers way more stable environment, regardless on the host machine runinng the agent. Notice there is no need to use custom containers, as the agent will basically run the venv process, only inside a container, allowing you to reuse offf the shelf containers.

If you were to add this, where would you put it? I can use a modified version of 

clearml-agent

Yep, that would be the place, basically you can add it here,
https://github.com/allegroai/clearml-agent/blob/bd411a19843fbb1e063b131e830a4515233bdf04/clearml_agent/commands/worker.py#L2095
You can get the actual script from the Task object current_task .
Please feel free to PR once you get it working, I'll be more than happy to help

If you were to add this, where would you put it? I can use a modified version of clearml-agent

Our code is tightly integrated with protobuffers which needs to be re-compiled every now and then. We have a script to do that. If that’s not done, some imports end up failing.

Unfortunately not yet in venv mode. What would you have put there?

AgitatedDove14 can I specify a script to be run after pip install packages is done? I see that it’s possible in docker mode.

Nice!

I think it works, I’m fixing something else that came up.

Ok!

First let's verify with the manual change, but yes

Great find! So a pip upgrade should fix it hopefully.

LazyTurkey38 I think this is caused by new versions of pip to report the wrong link:
https://github.com/bwoodsend/pip/commit/f533671b0ca9689855b7bdda67f44108387fe2a9

Try to manually edit the "Installed Packages" (right click the Task, select "reset", now you can edit the section)
and change it to :
-e git+ssh@github.com:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package(assuming " pip install -e mailto:git+ssh@github.com :user/... " will work, should solve the issue )

Is there a way to make it use ssh+git instead of git+git ? Maybe add a force_ssh_pip_install to the agent config?

The commit is valid for sure.

Well this doesn’t work
pip install -e

-e

:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package

Is this the correct link to the repo and a valid commit id ?

Can you post a few more lines from the agent's log ?
Something is failing to install I'm just not sure what

It is indeed autopopulated by init

... more-itertools==8.6.0 -e git+git@github.com:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package msgpack==1.0.2 msgpack-numpy==0.4.7.1 ...

The agent is installing the "Installed Paclages" section of the Task (think of it as requirements.txt)
And again, what do you have there? Is it the outcome of the Task.init auto populating it?

The private_package can be installed by doing pip install git+ssh://git@github.com/user/private_package.git but the agent is trying to do pip install private_package which won’t work.

Can you copy the "Installed Packages" here, and point to the package causing the issue?

I hadn’t enabled that line when the failure happened.

I am already forcing ssh auth

p.s. you should remove this line 🙂
extra_index_url: ["git@github.com:salimmj/xxxx"]

It doesn’t install it automatically, I think I need to specify it somewhere, see the above error. Or am I misunderstanding?

This means that in your "Installed packages" you should see the line:
Notice that this is not a pypi artifactory (i.e. a server to add to the extra index url for pip), this is a direct pip install from a git repository, hence it should be listed in the "installed packages".
If this is the way the package was installed locally, you should have had this line in the installed packages.
The clearml agent should take care of the authentication for you (specifically here, it should do nothing).
If you need all git references converted to SSH authentication (by default it is https with user/pass), set
force_git_ssh_protocol: true in your agent's clearml.conf file
https://github.com/allegroai/clearml-agent/blob/bd411a19843fbb1e063b131e830a4515233bdf04/docs/clearml.conf#L25

AgitatedDove14 this works: pip install git+ssh://git@github.com/user/repo.git

I’m not sure if 

https

 will work because I want to use ssh keys for creds.

BTW: I was not aware github provide pypi like artifactory, do they ?
Regrading SSH keys, they are passed from the host machine (i.e. in venv mode it will use the SSH keys from the user running the agent, and n docker mode, they are automatically mapped into the container)