looks promising. couple of questions:
wdym 'executed on different machines'? is there an mlclearish way of running a pipeline, ie something instead of implementing my own run method? i did my own run because i wanted to organize each pipeline into its own experiment folder and skip stages if they were already ran but it feels hacky and you folks have prolly a better way of doing this

AgitatedDove14 currently we use mlflow in some custom code to log and load artifacts

each child of Pipeline is a self contained pipeine, eg ModelPipeline. each step of the pipeline is a method, the order being set in the attribute array stage_handler_mapping . in the mlflow ui each stage, i.e. each methods results, is represented as a run within a fixed experiment

This means if you have a step that needs GPU it will be launched on a GPU machine vs steps that are cpu/logic. Make sense ?

yes, nice move. my question was to make sure that the steps are not run in parallel because each one builds upon the previous one

Maybe a few clearml example s would help?

id checked out that file but now with your explanation it is clear to me how to do it. so my takeaway is that if the funcs are class methods the decorators wont break, right? i had had a...

sure AgitatedDove14 . boiled down my pipeline into bare bones functionality and one file