Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey, What Is The Exact Difference Between

Hey, what is the exact difference between agent.package_manager.system_site_packages and trains-agent --install-globally ?

  
  
Posted 4 years ago
Votes Newest

Answers 10


Yes, thanks for your answer!

  
  
Posted 4 years ago

Since pytorch is a special example (the agent will pick the correct pytorch based on the installed CUDA) , the agent will first make sure the file is downloaded, and then pass the resolving for pip to decide if it necessary to install. (bottom line, we downloaded the torch for no reason but it is cached so no real harm done) It might be the second package needs a specific numpy version... this resolving is don't by pip, not the agent specifically. Anyhow --system-site-packages is applicable on a venv level, so no need to worry about that.Make sense ?

  
  
Posted 4 years ago

I hitted enter too fast ^^
Installing them globally via
$ pip install numpy opencv torch will install locally with warning:
Defaulting to user installation because normal site-packages is not writeable , therefore the installation will take place in ~/.local/lib/python3.6/site-packages , instead of the default one. Will this still be considered as global site-packages and still be included in experiments envs? From what I tested it does

  
  
Posted 4 years ago

Will this still be considered as 

global site-packages

This is a pip settings, I "think" it inherits from the local user's installation, but I would actually install with "sudo pip" that will definitely be "inherited"

  
  
Posted 4 years ago

Okay I think this is our answer 🙂

  
  
Posted 4 years ago

Hi JitteryCoyote63 ,
When using the --install-globally switch, the agent.package_manager.system_site_packages flag will be automatically be set to true .
While the system_site_packages flag simply controls how the agent configures pip (thus allowing you to use the agent in systems where you've already preinstalled some python packages), the --install-globally switch controls how the agent sets up virtual environments, and makes it install the required packages before creating the virtual environment, thus speeding up multiple venv creation (as the packages are only installed once) - doing that required using the system_site_packages flag, otherwise packages installed outside of the virtual environment won't be accessible...

  
  
Posted 4 years ago

Thanks for the clarification SuccessfulKoala55 ! A follow-up question:
I would like to install several packages (opencv, numpy, torch) in the system-site-packages so that they are available in each experiment (to reduce setup time of the experiments). Installing them globally via

  
  
Posted 4 years ago

It should work 🙂 as long as the versions match, if they don't the venv will install the version you need (which is great, only penalty is the install, download wise it will be cached)

  
  
Posted 4 years ago

I tested by installing flask in the default env -> which was installed in the ~/.local/lib/python3.6/site-packages folder. Then I created a venv with flag --system-site-packages . I activated the venv and flask was indeed available

  
  
Posted 4 years ago

AgitatedDove14 I now tested with a real experiment, it works, but I saw two issues:
It first doesnt detect torch, downloads it but then says that it is already installed so it doesn't install it. One of the dependency of my repository is another repository (repo-2 in the logs). Both my repositories require numpy . When installing the first repository, it says Requirement already satisfied: numpy in /home/workeruser/.local/lib/python3.6/site-packages . Correct. But then it says Installing collected packages: numpy, opencv-python, ... . So I guess when installing the package from the second repo it looses somehow the --system-site-packages param

  
  
Posted 4 years ago