Hi JitteryCoyote63 ,
When using the --install-globally
switch, the agent.package_manager.system_site_packages
flag will be automatically be set to true
.
While the system_site_packages
flag simply controls how the agent configures pip
(thus allowing you to use the agent in systems where you've already preinstalled some python packages), the --install-globally
switch controls how the agent sets up virtual environments, and makes it install the required packages before creating the virtual environment, thus speeding up multiple venv creation (as the packages are only installed once) - doing that required using the system_site_packages
flag, otherwise packages installed outside of the virtual environment won't be accessible...
Thanks for the clarification SuccessfulKoala55 ! A follow-up question:
I would like to install several packages (opencv, numpy, torch) in the system-site-packages
so that they are available in each experiment (to reduce setup time of the experiments). Installing them globally via
It should work 🙂 as long as the versions match, if they don't the venv will install the version you need (which is great, only penalty is the install, download wise it will be cached)
I hitted enter too fast ^^
Installing them globally via$ pip install numpy opencv torch
will install locally with warning:Defaulting to user installation because normal site-packages is not writeable
, therefore the installation will take place in ~/.local/lib/python3.6/site-packages
, instead of the default one. Will this still be considered as global site-packages
and still be included in experiments envs? From what I tested it does
Will this still be considered asÂ
global site-packages
This is a pip settings, I "think" it inherits from the local user's installation, but I would actually install with "sudo pip" that will definitely be "inherited"
I tested by installing flask in the default env -> which was installed in the ~/.local/lib/python3.6/site-packages
folder. Then I created a venv with flag --system-site-packages
. I activated the venv and flask was indeed available
AgitatedDove14 I now tested with a real experiment, it works, but I saw two issues:
It first doesnt detect torch, downloads it but then says that it is already installed so it doesn't install it. One of the dependency of my repository is another repository (repo-2 in the logs). Both my repositories require numpy
. When installing the first repository, it says Requirement already satisfied: numpy in /home/workeruser/.local/lib/python3.6/site-packages
. Correct. But then it says Installing collected packages: numpy, opencv-python, ...
. So I guess when installing the package from the second repo it looses somehow the --system-site-packages
param
Since pytorch is a special example (the agent will pick the correct pytorch based on the installed CUDA) , the agent will first make sure the file is downloaded, and then pass the resolving for pip to decide if it necessary to install. (bottom line, we downloaded the torch for no reason but it is cached so no real harm done) It might be the second package needs a specific numpy version... this resolving is don't by pip, not the agent specifically. Anyhow --system-site-packages is applicable on a venv level, so no need to worry about that.Make sense ?