Hi TenseOstrich47 ,

You can treat the “installed packages” part on your cloned task the same as a “requirements.txt” file, just add:
git+ https://github.com/ or git+ssh://github.com/ to the packages you like to get.

Notice you will need the credentials to those packages

Never mind. I think I figured it out. Thanks for your help 🙂

TenseOstrich47 FYI:
This might what you are looking for 🙂
https://github.com/allegroai/clearml-agent/blob/822984301889327ae1a703ffdc56470ad006a951/docs/clearml.conf#L61

Hi TenseOstrich47
Does the .ssh folder on the user running the agent contain the correct credentials ?
Basically from the user running the agent on the agent's machine can you clone the repo with:
ssh://git@github.com/15gifts/py-db.git

Rightttttt I think I am starting to understanding the architecture now lol. Thank you so much for your help!

Hey guys. Installing from the private repo is still failing. We have added the relevant deploy key to the repo, but I still get an error when trying to clone and install. Any ideas?

This is what I see under the 'Installed Packages' section
` # Python 3.8.5 (default, Sep 4 2020, 02:22:02) [Clang 10.0.0 ]
azure_storage_blob == 12.6.0
boto3 == 1.11.17
clearml == 0.17.4
git+ssh://git@github.com/15gifts/py-db.git

Detailed import analysis

**************************

IMPORT PACKAGE azure_storage_blob

clearml.storage: 0

IMPORT PACKAGE boto3

clearml.storage: 0

IMPORT PACKAGE clearml

pre-engine-traits-pipeline.ipynb: 11

IMPORT PACKAGE py_db

pre-engine-traits-pipeline.ipynb: 13 `

That is a neat way of making it work! Thanks Martin. Once I've added the SSH key to the deployment keys in that repo, then the change in the config should work right? I'm guessing the extra index URL can be a URL to the github repo of interest? (not another privately hosted pypi repo)

Hey Martin. We have managed to resolve this. FYI the issue was with the resolving of the host. It had to be changed from @github.com to what the host is in the ssh config file!

Yeah I would say that a demo on this would be great. I think this task is difficult as is given the differences in deployment architectures, but for common tasks it would be good to have some additional docs/examples 🙂

My pleasure 🙂
Maybe we should do a webinar... I have a feeling the MLOps aspects are not as straight forward as we would like to think ...

👍

TenseOstrich47

I noticed that with one agent, only one task gets executed at one time

Yes you can 🙂
Also, you are correct, a single agent will run a single Task at a time, that said you can have multiple agents running on the same machine, and when you launch them you specify which GPUs they use (in theory they can share the same GPU, but your code might not like it 😉 )
You can see a few examples here:
https://github.com/allegroai/clearml-agent#running-the-clearml-agent

TenseOstrich47 make sense 🙂

I'm guessing the extra index URL can be a URL to the github repo of interest?

The extra index URL is exactly what you would be passing to pip install, meaning it has to comply to pypi artifactory api.
Make sense ?

While we are here - excuse my ignorance for now if this has already been stated in the docs ..

Is it possible to launch multiple clearml-agents on a dedicated clearml-agent server? I noticed that with one agent, only one task gets executed at one time

If you are referring to Python packages, you should add it to the “installed packages” section on the execution tab of you experiment. just replace the version with your package location, for example, if you have clearml == 0.17.4 , you can replace it with
git+to get the latest version of clearml.

Hey Alon. Thanks for the response. I'm not quite sure I follow your answer. Where do I add this git+ssh and how?

Here is the error message from the console
Collecting git+ssh://****@github.com/15gifts/py-db.git Cloning ssh://****@github.com/15gifts/py-db.git to /tmp/pip-req-build-xai2xts_ Running command git clone -q 'ssh://****@github.com/15gifts/py-db.git' /tmp/pip-req-build-xai2xts_ ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.