I'm guessing the extra index URL can be a URL to the github repo of interest?
The extra index URL is exactly what you would be passing to pip install, meaning it has to comply to pypi artifactory api.
Make sense ?
Never mind. I think I figured it out. Thanks for your help 🙂
Rightttttt I think I am starting to understanding the architecture now lol. Thank you so much for your help!
While we are here - excuse my ignorance for now if this has already been stated in the docs ..
Is it possible to launch multiple clearml-agents on a dedicated clearml-agent server? I noticed that with one agent, only one task gets executed at one time
Hey Martin. We have managed to resolve this. FYI the issue was with the resolving of the host. It had to be changed from @github.com to what the host is in the ssh config file!
Hey guys. Installing from the private repo is still failing. We have added the relevant deploy key to the repo, but I still get an error when trying to clone and install. Any ideas?
This is what I see under the 'Installed Packages' section
` # Python 3.8.5 (default, Sep 4 2020, 02:22:02) [Clang 10.0.0 ]
azure_storage_blob == 12.6.0
boto3 == 1.11.17
clearml == 0.17.4
git+ssh://git@github.com/15gifts/py-db.git
Detailed import analysis
**************************
IMPORT PACKAGE azure_storage_blob
clearml.storage: 0
IMPORT PACKAGE boto3
clearml.storage: 0
IMPORT PACKAGE clearml
pre-engine-traits-pipeline.ipynb: 11
IMPORT PACKAGE py_db
pre-engine-traits-pipeline.ipynb: 13 `
Hi TenseOstrich47 ,
You can treat the “installed packages” part on your cloned task the same as a “requirements.txt” file, just add:
git+ https://github.com/ or git+ssh://github.com/ to the packages you like to get.
Notice you will need the credentials to those packages
TenseOstrich47
I noticed that with one agent, only one task gets executed at one time
Yes you can 🙂
Also, you are correct, a single agent will run a single Task at a time, that said you can have multiple agents running on the same machine, and when you launch them you specify which GPUs they use (in theory they can share the same GPU, but your code might not like it 😉 )
You can see a few examples here:
https://github.com/allegroai/clearml-agent#running-the-clearml-agent
Yeah I would say that a demo on this would be great. I think this task is difficult as is given the differences in deployment architectures, but for common tasks it would be good to have some additional docs/examples 🙂
Hi TenseOstrich47
Does the .ssh folder on the user running the agent contain the correct credentials ?
Basically from the user running the agent on the agent's machine can you clone the repo with:ssh://git@github.com/15gifts/py-db.git
My pleasure 🙂
Maybe we should do a webinar... I have a feeling the MLOps aspects are not as straight forward as we would like to think ...
Here is the error message from the consoleCollecting git+ssh://****@github.com/15gifts/py-db.git Cloning ssh://****@github.com/15gifts/py-db.git to /tmp/pip-req-build-xai2xts_ Running command git clone -q 'ssh://****@github.com/15gifts/py-db.git' /tmp/pip-req-build-xai2xts_ ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.
Hey Alon. Thanks for the response. I'm not quite sure I follow your answer. Where do I add this git+ssh and how?
TenseOstrich47 FYI:
This might what you are looking for 🙂
https://github.com/allegroai/clearml-agent/blob/822984301889327ae1a703ffdc56470ad006a951/docs/clearml.conf#L61
That is a neat way of making it work! Thanks Martin. Once I've added the SSH key to the deployment keys in that repo, then the change in the config should work right? I'm guessing the extra index URL can be a URL to the github repo of interest? (not another privately hosted pypi repo)
If you are referring to Python packages, you should add it to the “installed packages” section on the execution tab of you experiment. just replace the version with your package location, for example, if you have clearml == 0.17.4 , you can replace it withgit+to get the latest version of clearml.