UptightCoyote42 nice!
BTW: make sure to clear requirements["conda"]
(not visible on the UI but tells the agent which packages were used by conda, out effort to try and see if we could do pip/conda interoperability , not sure if it actually paid off 🙂

Not sure if this is a valid solution, but if you have your docker container already pre-configured with all requirements - you can set the Task to ignore/disregards your DEV requirements. It should look somewhat like this -

` def ignore_task_requirements():
task=Task.current_task()
with task._edit_lock:
task.reload()
script = task.data.script

    pip_req = script.requirements["pip"]
    pip_req_splitted = pip_req.split("\n")
    pip_req_commented = '\n#'.join(pip_req_splitted)
    script.requirements["pip"] = pip_req_commented

    task._edit(script=script) `

SmarmySeaurchin8 we are looking into " https://github.com/conda/conda-pack " , allowing you to store an entire conda environment into a tar.gz file. With this feature, we could utilize the "base docker image" as a "base conda environment" basically deploying a fully stored conda env on the agent machine.
One could think of it as a poor wo/man's docker solution. That said it does have some merits like, bare-metal connection on Windows/Mac that is missing from the docker solution...

SmarmySeaurchin8 Following up on ColossalDeer61 's hint, notice https://allegroai-trains.slack.com/archives/CTK20V944/p1597248476076700?thread_ts=1597248298.075500&cid=CTK20V944 not-too-old thread on reusing globally installed packages.

You probably know you don't have to use docker with your agent. The alternative is to use "ad hoc" virtual environments.

It is a bit tricky, you need to remove the requirements from the queued task configuration. But you can't remove them all, since in that case the agent will use your project's requirements file (if I remember correctly).
I simply kept a single "light weight" requirement in the list, just to avoid looking for my requirements list.
And you need to set a flag for the agent to inherit system installations. In this mode the agent creates new virtual environment, but can use all the system installations.