LazyTurkey38 I think this is caused by new versions of pip to report the wrong link:
https://github.com/bwoodsend/pip/commit/f533671b0ca9689855b7bdda67f44108387fe2a9
The agent is installing the "Installed Paclages" section of the Task (think of it as requirements.txt)
And again, what do you have there? Is it the outcome of the Task.init auto populating it?
AgitatedDove14 can I specify a script to be run after pip install packages is done? I see that it’s possible in docker mode.
I see..
Generally speaking If that is the case, I would think it might be better to use the docker mode, it offers way more stable environment, regardless on the host machine runinng the agent. Notice there is no need to use custom containers, as the agent will basically run the venv process, only inside a container, allowing you to reuse offf the shelf containers.
If you were to add this, where would you put it? I can use a modified version of
clearml-agent
Yep, that would be the place, basically you can add it here,
https://github.com/allegroai/clearml-agent/blob/bd411a19843fbb1e063b131e830a4515233bdf04/clearml_agent/commands/worker.py#L2095
You can get the actual script from the Task object current_task
.
Please feel free to PR once you get it working, I'll be more than happy to help
Great find! So a pip upgrade should fix it hopefully.
First let's verify with the manual change, but yes
Our code is tightly integrated with protobuffers which needs to be re-compiled every now and then. We have a script to do that. If that’s not done, some imports end up failing.
p.s. you should remove this line 🙂extra_index_url: ["git@github.com:salimmj/xxxx"]
AgitatedDove14 this works: pip install
git+ssh://git@github.com/user/repo.git
Try to manually edit the "Installed Packages" (right click the Task, select "reset", now you can edit the section)
and change it to :-e git+ssh@github.com:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package
(assuming " pip install -e
mailto:git+ssh@github.com :user/...
" will work, should solve the issue )
-e
:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package
Is this the correct link to the repo and a valid commit id ?
Can you post a few more lines from the agent's log ?
Something is failing to install I'm just not sure what
I think it works, I’m fixing something else that came up.
I hadn’t enabled that line when the failure happened.
I wonder if using our own containers which should have most the deps will work better than a simpler container.
Why not, it's transparent, just run in --docker mode and provide a default docker image if the Task doesn't specify one.
I’m not sure if
https
will work because I want to use ssh keys for creds.
BTW: I was not aware github provide pypi like artifactory, do they ?
Regrading SSH keys, they are passed from the host machine (i.e. in venv mode it will use the SSH keys from the user running the agent, and n docker mode, they are automatically mapped into the container)
... more-itertools==8.6.0 -e git+git@github.com:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package msgpack==1.0.2 msgpack-numpy==0.4.7.1 ...
This means that in your "Installed packages" you should see the line:
Notice that this is not a pypi artifactory (i.e. a server to add to the extra index url for pip), this is a direct pip install from a git repository, hence it should be listed in the "installed packages".
If this is the way the package was installed locally, you should have had this line in the installed packages.
The clearml agent should take care of the authentication for you (specifically here, it should do nothing).
If you need all git references converted to SSH authentication (by default it is https with user/pass), setforce_git_ssh_protocol: true
in your agent's clearml.conf file
https://github.com/allegroai/clearml-agent/blob/bd411a19843fbb1e063b131e830a4515233bdf04/docs/clearml.conf#L25
If you were to add this, where would you put it? I can use a modified version of clearml-agent
The private_package
can be installed by doing pip install
git+ssh://git@github.com/user/private_package.git but the agent is trying to do pip install private_package
which won’t work.
If venv works inside containers that’s even better. We actually have custom containers that build on master merges. I wonder if using our own containers which should have most the deps will work better than a simpler container.
Is there a way to make it use ssh+git
instead of git+git
? Maybe add a force_ssh_pip_install
to the agent config?
It doesn’t install it automatically, I think I need to specify it somewhere, see the above error. Or am I misunderstanding?
Unfortunately not yet in venv mode. What would you have put there?
Can you copy the "Installed Packages" here, and point to the package causing the issue?