I'm new here, hello everyone 🙂
I have a question about pipelines. In my project, I try to run some code doing stuff, via a ClearML pipeline, in a ClearML agent/ docker mode.
clearml-agent daemon --docker registry/my_image
I need the docker image because do_stuff()
needs specific things in the image as dependencies.
I run the clearml-agent on my own computer.
The code launching the pipeline looks like this:
# my_proj/pipeline.py
from my_module.stuff import do_stuff
from clearml import PipelineController
pipe = PipelineController(
...
)
pipe.add_function_step(function=do_stuff)
pipe.start()
When I run that code on my own computer, I can see the pipeline ongoing on ClearML server. The first step begins and fails with:
Traceback (most recent call last):
File "/root/.clearml/venvs-builds/3.8/code/pipeline.py", line 5, in <module>
from my_module import do_stuff
ModuleNotFoundError: No module named 'my_module'
Now I wonder how the clearml agent is supposed to find that code.
In my understanding, clearml agent creates a venv inside the docker container, installing clearml stuff and then run pipeline.py
(which I don't know how it appears here). It doesn't seem to git clone
anything.
I need your help to understand that.
Should the code be embedded in the docker image? Any alternative? (map volumes, ...) How to set path to code to execute ?
Thanks