wdym 'executed on different machines'?The assumption is that you have machines (i.e. clearml-agents) connected to clearml, which would be running all the different components of the pipeline. Think out of the box scale-up. Each component will become a standalone Job and the data will be passed (i.e. stored and loaded) automatically on the clearml-server (can be configured to be external object storage as well). This means if you have a step that needs GPU it will be launched on a GPU machine vs steps that are cpu/logic. Make sense ?
is there an mlclearish way of running a pipeline, ie something instead of implementing my own run method? i
What do you mean by "i did my own run because i wanted" ? Maybe a few clearml example s would help?
https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py
Does that help?