Hi,
I'm working on our ML project using ClearML for pipeline management. I have a separate function for data preparation that I'd like to use in the ClearML pipeline. To keep the pipeline script clean, I prefer not to define this function directly within the pipeline script.
Is there a way to reference and execute this external data preparation function in the ClearML pipeline without embedding it in the main script? Any suggestions or best practices for maintaining separation of concerns while still allowing ClearML to access and run the function would be greatly appreciated.
Thanks in advance for your help!
Hi @<1669152726245707776:profile|ManiacalParrot65> ,
Yes, you can wrap the separate function with a decorator so the function will run as a separate step in the pipeline, and even can cache the step for multi runs.
You can also add the function without a decorator, as a step to the pipeline with PipelineController.add_function_step() .
You can read about it here (with some examples too).
