Or maybe you could bundle some parameters that belongs to PipelineDecorator.component into high-level configuration variable (something like PipelineDecorator.global_config (?))
So in the PipelineController we have a per step callback and generic callbacks (i.e. for all the steps), is this what you are referring to ?
Well, I can see the difference here. Using the new pipelines generation the user has the flexibility to play with the returned values of each step.
Yep 🙂
We can process those values before passing them to the next step
Also correct 🙂
so maybe makes little sense to include those callbacks in this case
That was my thinking... why go with cumbersome callbacks, when you have the ability to write it as part of the execution logic
BTW: I haven't thought about the exception catching part, and I'll make sure we add PipelineDecorator.get_current_pipeline()
this way you could do:@PipelineDecorator.pipeline(name='custom pipeline logic', project='examples', version='0.0.5') def executing_pipeline(pickle_url, mock_parameter='mock'): print('launch step two') try: processed_data = step_two(data_frame) except Exception as e: # stop all running pipeline steps PipelineDecorator.get_current_pipeline().stop() raise