For anyone following along, my lesson was configuring the clearml-agent daemon with the --docker flag to instruct it to spawn tasks in containers (and using the docker arg passed through to my Pipeline component)

Right, my only complaint is it appears to be using cached wheels and building them (for packages like numpy , scipy , etc) even though numpy is available in the Python runtime env

Thanks, my pipeline script only takes a dependency on clearml as well as an internal library (local Python module installed into the Docker image) that provides the _train_and_evaluate function as seen above

Thank you! I adjusted my pipeline logic so that the component used packages=[]

Funny enough I’m running into a new issue now. Does this mean I need to configure the Agent’s runtime environment so it has the necessary dependencies to execute Pipeline script?
` # Agent Logs
Starting Task Execution:

Traceback (most recent call last):
File "/Users/developer/.clearml/venvs-builds/3/code/train_and_evaluate.py", line 1, in <module>
from clearml import Task, TaskTypes
ModuleNotFoundErro...

Pretty standard global install

https://gist.github.com/stevenhoelscher/0d345e26630e7d16ab76802871c39bd5

Just threw a new file into the gist above

It doesn’t look like it even gets to the point where it installs from the numpy wheel (because it errors out installing Pillow elsewhere).

What’s interesting to me (as a ClearML newbie) is it’s clearly compiling that wheel using my host machine (MacOS).

I would have expected there to be separation between the “pipeline runner” if you will and the task. I would expect the pipeline runner to only need a dependency on ClearML and for the task to be sp...