Unanswered
I Have A Pipeline With Tasks A->B->C.
I Want To Be Able To Trigger It Manually, And Skip A Regardless Of It’S Cache Status.
I Want To Pass B Value That Represents A’S Output If Needed.
What’S A Good Way To Achieve This (Can Be Ui-Based, Or Pipeline-Gymnas
AgitatedDove14
Sort of.
I would go with something which is more like:
` execution_plan = {'step_b':'b_result', step_c: None, ...}
@PipelineDecorator.pipeline(...)
def pipeline(execution_plan):
step_results = {}
for step in pipeline.get_dag():
if step.name in execution_plan.keys():
step_results[step.name] = execution_plan[step.name] or step(**step_results)
`The ‘execution plan’ specifies list of steps to run (keys) and for each, whether we should use a user-specified value.
Then we iterate on the steps in the dag.
If the step is not at all in the execution plan, skip it entirely.
If the step is in the execution plan and has a user-provided value - use that value.
If it doesn’t have a user provided value - execute the step with all **kwargs it exects.
if decided to execute - fall back on clearml behaviour (including cache configuration)
169 Views
0
Answers
2 years ago
one year ago