Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
How Do People Solve This? If I Am Pip Installing A Custom Package From .Tar.Gz, How Can I Ensure That If I Run The Experiment (Initially Run From A Notebook) Via The Queueing It Can Be Properly Installed Steps - Notebook -> Get A Tar.Gz From S3 -> Pip I

How do people solve this?

If i am pip installing a custom package from .tar.gz, how can i ensure that if i run the experiment (initially run from a notebook) via the queueing it can be properly installed

Steps -
notebook -> get a tar.gz from s3 -> pip install tar.gz
now I want to schedule this on a worker amd ensure that the package is installed there as well

I can see that a pattern is use system site packages. Is conda env (like those available in Sagemaker notebook instances) possible?

Another seems to be docker base image and let worker build from that?

  
  
Posted 3 years ago
Votes Newest

Answers 12


What's the pip verison?

  
  
Posted 3 years ago

TrickySheep9 is this a conda package or a wheel you are installing manually ?

  
  
Posted 3 years ago

The minerva one is my custom package AgitatedDove14

  
  
Posted 3 years ago

Very bad at python packaging terms - but I use pyproject.toml and python -m build that generates a wheel and a tar and I install the tar

  
  
Posted 3 years ago

AgitatedDove14 - apologies for late reply. So to give context this in a Sagemaker notebook which has conda envs.

I use a lifecycle like this to pip install a package (a .tar.gz downloaded from s3) in a conda env- https://github.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/blob/master/scripts/install-pip-package-single-environment/on-start.sh

In the notebook I can do things like create experiments and so on. Now the problem is in running the cloned experiment and also for pipelines running remotely etc.

  
  
Posted 3 years ago

If i were to push the private package to, say artifactory, is it possible to use that do the install?

  
  
Posted 3 years ago

If i were to push the private package to, say artifactory, is it possible to use that do the install?

Yes that's the recommended way 🙂
You add the private repo here, for the agent to use:
https://github.com/allegroai/clearml-agent/blob/e93384b99bdfd72a54cf2b68b3991b145b504b79/docs/clearml.conf#L65

  
  
Posted 3 years ago

# Python 3.6.13 | packaged by conda-forge | (default, Feb 19 2021, 05:36:01) [GCC 9.3.0] argparse == 1.4.0 boto3 == 1.17.70 minerva == 0.1.0 torch == 1.7.1 torchvision == 0.8.2

  
  
Posted 3 years ago

and I install the tar

I think the only way to do that is add it into the docker bash setup script (this is a bash script executed before Task)

  
  
Posted 3 years ago

When you install using pip <filename> you should end up with something like:
minerva @ file://... or minerva @ https://...

  
  
Posted 3 years ago

Could you send the "installed packages" section of the Task that was created in the notebook ?

  
  
Posted 3 years ago

Ohh is this a conda installed ?

  
  
Posted 3 years ago