RotundHedgehog76 , Hi 🙂
In none-docker mode, the agent simply runs on the target machine and creates a python virtual environment and runs the experiment in it. This means that you need to handle CUDA and other system installations for the agent to run your experiments.
When agent runs in docker mode, it will set up a docker image (that was pre-defined) and will create the virtual environment in that docker.
Basically the docker option gives you the most surgical and exactly pre-defined system to run it 🙂
Does that help?
Just to emphasize, pip and docker are not different modes, and are not interchangeable - when running in docker mode, the agent will still create a virtual environment and install python packages in it using the selected package manager - pip, conda etc.
In the UI when looking at a task under 'execution' tab, if you scroll down you have a 'container' option. If you define an image there the agent will use it if you use the '--docker' argument
You can specify a different docker image per experiment, so the same agent can run many different docker images (As long as it is run in docker mode from the start) 🙂
CostlyOstrich36 this sounds great. How do I accomplish that?
This means that an agent only ever spins up one particular image? I'd like to define different container images for different tasks, possibly even build them in the process of starting a task. Is such a thing possible?