Unanswered
Hi All! I Have A Question About Pipelines. My Pipeline Consists Of Several Steps:
Yes I think it absolutely fine. Here is the pseudocode of my understanding with ClearML syntax:
`
def complex_steps(args):
As far as I see the functions should be implemented inside the step for ClearML be able to see them
@sub_node
def action_1(params):
....
return result
@sub_node
def action_2(params):
....
return result
@sub_node
def action_3(params_1, params_2):
....
return result
act1_result = action_1(args.param1)
act2_result = action_2(args.param2)
return action_3(act1_result, act2_result)
As a result of this function in clearML will be
complex_steps |
---------------------------------- |
pipe_c = PipelineController(strategy=single_agent)
pipe_c.add_functional_step(complex_steps, strategy=pipe_c.agent, outputs=[path_to_big_dataset])
pipe_c.add_functional_step(complex_steps2, strategy=pipe_c.agent, func_kwargs={paths=${complex_steps.path_to_big_dataset})
pipe_c.add_functional_step(complex_steps3, strategy=default)
dafault strategy means – use different pod for step `I hope it does make some sense 🙂
150 Views
0
Answers
2 years ago
one year ago