Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All! I Noticed When A Pipeline Fails, All Its Components Continue Running. Wouldn'T It Make More Sense For The Pipeline To Send An Abort Signal To All Tasks That Depend On The Pipeline? I'M Using Clearml V1.1.3Rc0 And Clearml-Agent 1.1.0

Hi all!
I noticed when a pipeline fails, all its components continue running. Wouldn't it make more sense for the pipeline to send an abort signal to all tasks that depend on the pipeline?
I'm using clearml v1.1.3rc0 and clearml-agent 1.1.0

  
  
Posted 2 years ago
Votes Newest

Answers 21


Or maybe you could bundle some parameters that belongs to PipelineDecorator.component into high-level configuration variable (something like PipelineDecorator.global_config (?))

So in the PipelineController we have a per step callback and generic callbacks (i.e. for all the steps), is this what you are referring to ?

Well, I can see the difference here. Using the new pipelines generation the user has the flexibility to play with the returned values of each step.

Yep 🙂

We can process those values before passing them to the next step

Also correct 🙂

so maybe makes little sense to include those callbacks in this case

That was my thinking... why go with cumbersome callbacks, when you have the ability to write it as part of the execution logic
BTW: I haven't thought about the exception catching part, and I'll make sure we add PipelineDecorator.get_current_pipeline() this way you could do:
@PipelineDecorator.pipeline(name='custom pipeline logic', project='examples', version='0.0.5') def executing_pipeline(pickle_url, mock_parameter='mock'): print('launch step two') try: processed_data = step_two(data_frame) except Exception as e: # stop all running pipeline steps PipelineDecorator.get_current_pipeline().stop() raise

  
  
Posted 2 years ago

GiganticTurtle0 My apologies, I made a mistake, this will not work 😞
In the example above "step_two" is executed "instantaneously" , meaning it is just launching the remote task, it is not actually waiting for it.
This means an exception will not be raised in the "correct" context (actually it will be raised in a background thread).
That means that I think we have to have a callback function, otherwise there is no actual way to catch the failed pipeline task.
Maybe the only real thing is just an argument for "abort_on_failed_step" ? WDYT?

  
  
Posted 2 years ago

I'm totally agree with the pipelinecontroller/decorator part. Regarding the proposal for the component parameter, I also think it would be a good feature, although it might mislead the fact that there will be times when the pipeline will fail because it is an intrinsically crucial step, so it doesn't matter whether 'continue_pipeline_on_failure' is set to True or False. Anyway, I can't think a better way to deal with that right now.

  
  
Posted 2 years ago

Okay so my thinking is, on the pipelinecontroller / decorator we will have:
abort_all_running_steps_on_failure=False (if True, on step failing it will abort all running steps and leave)
Then per step / component decorator we will have
continue_pipeline_on_failure=False (if True, on step failing, the rest of the pipeline dag will continue)
GiganticTurtle0 wdyt?

  
  
Posted 2 years ago

Sure! That would be great too

  
  
Posted 2 years ago

GiganticTurtle0

That definitely makes sense. Where can I specify callbacks in the 

PipelineDecorator

 API?

Hmm there isn't one actually... (the interface I was thinking about was PipelineConroller ...)
Would it make sense to throw an exception in the pipeline execution code?

BTW: I just verified, if the pipeline step fails an exception is raised (ValueError)

  
  
Posted 2 years ago

However, are you thinking of including this callbacks features in the new pipelines as well?

Can you see a good use case ? (I mean the infrastructure supports it, but sometimes too many arguments is just confusing, no?!)

  
  
Posted 2 years ago

Well, I see the same utility as it has in the first pipelines generation. After all, isn't the new decorator about keeping the same functionality but saving the user some boilerplate code?

  
  
Posted 2 years ago

Or maybe you could bundle some parameters that belongs to PipelineDecorator.component into high-level configuration variable (something like PipelineDecorator.global_config (?))

  
  
Posted 2 years ago

AgitatedDove14

  
  
Posted 2 years ago

The scheme is similar to the following:
main_pipeline (PipelineDecorator.pipeline) | |----------------------------------| | | inference_orchestrator_1 inference_orchestrator_2 (PipelineDecorator.component, (PipelineDecorator.component, acting as a pipeline) acting as a pipeline) | | | | data_preprocessing_1 data_preprocessing_2 (PipelineDecorator.component) (PipelineDecorator.component) | | | | inference_1 inference_2 (PipelineDecorator.component) (PipelineDecorator.component) | | | | output_predictions_1 output_predictions_2So if any step corresponding to 'inference_orchestrator_1' fails, then 'inference_orchestrator_2' keeps running. Is there any way to interrupt the main pipeline in case of failure of any of the inference pipelines?

  
  
Posted 2 years ago

So if any step corresponding to 'inference_orchestrator_1' fails, then 'inference_orchestrator_2' keeps running.

GiganticTurtle0 I'm not sure it makes sense to halt the entire pipeline if one step fails.
That said, how about using the post_execution callback, then check if the step failed, you could stop the entire pipeline (and any running steps), what do you think?

  
  
Posted 2 years ago

In my use case I have a pipeline that executes inference tasks with several models simultaneously. Each inference task is actually a component that acts as a pipeline, since it executes the required steps to generate the predictions (dataset creation, preprocessing and prediction). For this, I'm using the new pipeline functionality ( PipelineDecorator )

  
  
Posted 2 years ago

Hmm, I'm not 100% sure I follow. you have multiple models doing predictions. Is there a single data source that feeds to them and they run in parallel. or is one's output is another input and they run serially?

  
  
Posted 2 years ago

Are you talking about consecutive pipeline steps? Or parallel?

  
  
Posted 2 years ago

Or perhaps the complementary scenario with a continue_on_failed_steps parameter which may be a list containing only the steps that can be ignored in case of failure.

  
  
Posted 2 years ago

I think it could be a convenient approach. The new parameter abort_on_failed_steps could be a list containing the name of the steps for which the pipeline will stop its execution if any of them fail (so that we can ignore other steps that are not crucial to continue the pipeline execution)

  
  
Posted 2 years ago

Sure! That definitely makes sense. Where can I specify callbacks in the PipelineDecorator API?

  
  
Posted 2 years ago

Well, I can see the difference here. Using the new pipelines generation the user has the flexibility to play with the returned values of each step. We can process those values before passing them to the next step, so maybe makes little sense to include those callbacks in this case

  
  
Posted 2 years ago

Okey, so I could signal to the main pipeline the exception raised in any of the pipeline components and it should halt the whole pipeline. However, are you thinking of including this callbacks features in the new pipelines as well?

  
  
Posted 2 years ago

The new parameter 

abort_on_failed_steps

 could be a list containing the name of the

I like that, we can also have it as an argument per step (i.e. the decorator can say, abort_pipeline_on_fail or continue_pipeline_processing)

  
  
Posted 2 years ago
602 Views
21 Answers
2 years ago
one year ago
Tags