Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey All. Another Question - How Are Private Packages Handled/Installed So That Clearml-Agent Can Execute A Task? I Have A Bunch Of Private Repos For Communicating With The Data Warehouse. I Could Do A System-Wide Installation For It On The Clearml-Agent I

Hey all. Another question - how are private packages handled/installed so that clearml-agent can execute a task? I have a bunch of private repos for communicating with the data warehouse. I could do a system-wide installation for it on the clearml-agent instance but was wondering if there is a neater way of achieving this (especially if clearml-agent is deployed within a container)?

  
  
Posted 3 years ago
Votes Newest

Answers 18


Hi TenseOstrich47 ,

You can treat the “installed packages” part on your cloned task the same as a “requirements.txt” file, just add:
git+ https://github.com/ or git+ssh://github.com/ to the packages you like to get.

Notice you will need the credentials to those packages

  
  
Posted 3 years ago

TenseOstrich47

I noticed that with one agent, only one task gets executed at one time

Yes you can 🙂
Also, you are correct, a single agent will run a single Task at a time, that said you can have multiple agents running on the same machine, and when you launch them you specify which GPUs they use (in theory they can share the same GPU, but your code might not like it 😉 )
You can see a few examples here:
https://github.com/allegroai/clearml-agent#running-the-clearml-agent

  
  
Posted 3 years ago

If you are referring to Python packages, you should add it to the “installed packages” section on the execution tab of you experiment. just replace the version with your package location, for example, if you have clearml == 0.17.4 , you can replace it with
git+to get the latest version of clearml.

  
  
Posted 3 years ago

My pleasure 🙂
Maybe we should do a webinar... I have a feeling the MLOps aspects are not as straight forward as we would like to think ...

  
  
Posted 3 years ago

Hey Alon. Thanks for the response. I'm not quite sure I follow your answer. Where do I add this git+ssh and how?

  
  
Posted 3 years ago

I'm guessing the extra index URL can be a URL to the github repo of interest?

The extra index URL is exactly what you would be passing to pip install, meaning it has to comply to pypi artifactory api.
Make sense ?

  
  
Posted 3 years ago

Never mind. I think I figured it out. Thanks for your help 🙂

  
  
Posted 3 years ago

TenseOstrich47 make sense 🙂

  
  
Posted 3 years ago

While we are here - excuse my ignorance for now if this has already been stated in the docs ..

Is it possible to launch multiple clearml-agents on a dedicated clearml-agent server? I noticed that with one agent, only one task gets executed at one time

  
  
Posted 3 years ago

Yeah I would say that a demo on this would be great. I think this task is difficult as is given the differences in deployment architectures, but for common tasks it would be good to have some additional docs/examples 🙂

  
  
Posted 3 years ago

Rightttttt I think I am starting to understanding the architecture now lol. Thank you so much for your help!

  
  
Posted 3 years ago

👍

  
  
Posted 3 years ago

Here is the error message from the console
Collecting git+ssh://****@github.com/15gifts/py-db.git Cloning ssh://****@github.com/15gifts/py-db.git to /tmp/pip-req-build-xai2xts_ Running command git clone -q 'ssh://****@github.com/15gifts/py-db.git' /tmp/pip-req-build-xai2xts_ ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

  
  
Posted 3 years ago

Hi TenseOstrich47
Does the .ssh folder on the user running the agent contain the correct credentials ?
Basically from the user running the agent on the agent's machine can you clone the repo with:
ssh://git@github.com/15gifts/py-db.git

  
  
Posted 3 years ago

Hey Martin. We have managed to resolve this. FYI the issue was with the resolving of the host. It had to be changed from @github.com to what the host is in the ssh config file!

  
  
Posted 3 years ago

Hey guys. Installing from the private repo is still failing. We have added the relevant deploy key to the repo, but I still get an error when trying to clone and install. Any ideas?

This is what I see under the 'Installed Packages' section
` # Python 3.8.5 (default, Sep 4 2020, 02:22:02) [Clang 10.0.0 ]
azure_storage_blob == 12.6.0
boto3 == 1.11.17
clearml == 0.17.4
git+ssh://git@github.com/15gifts/py-db.git

Detailed import analysis

**************************

IMPORT PACKAGE azure_storage_blob

clearml.storage: 0

IMPORT PACKAGE boto3

clearml.storage: 0

IMPORT PACKAGE clearml

pre-engine-traits-pipeline.ipynb: 11

IMPORT PACKAGE py_db

pre-engine-traits-pipeline.ipynb: 13 `

  
  
Posted 3 years ago

That is a neat way of making it work! Thanks Martin. Once I've added the SSH key to the deployment keys in that repo, then the change in the config should work right? I'm guessing the extra index URL can be a URL to the github repo of interest? (not another privately hosted pypi repo)

  
  
Posted 3 years ago
1K Views
18 Answers
3 years ago
one year ago
Tags