Hi, I'M Currently Working With Clearml Pipelines And Would Like To Clarify Whether It'S Officially Supported To

Answered

Hi, I'm currently working with ClearML Pipelines and would like to clarify whether it's officially supported to invoke a sub-pipeline from within another pipeline , especially when running multiple variations in parallel. Is it possible to call one pipeline (e.g., sub_pipeline ) from within another pipeline (e.g., main_pipeline ) while:

Not manually specifying a base_task_id
Running multiple sub-pipelines in parallel
Collecting the results after all sub-pipelines finishWhen executing this, we encounter the following error: ValueError: Node 'sub_pipeline', base_task_id is empty

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					LonelyKangaroo55
				
					0
					 × 1

Votes Newest

Answers 2

Hi @<1603198163143888896:profile|LonelyKangaroo55> , I don't believe this is supported.

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

@PipelineDecorator.pipeline(
name="Sub Pipeline",
project="Pipelines",
version="1.0",
multi_instance_support="parallel",
)
def sub_pipeline(parameter):
print(f"Running sub-pipeline with parameter={parameter}")
return parameter * 2

@PipelineDecorator.pipeline(
name="Main Pipeline",
project="Pipelines",
version="1.0",
)
def main_pipeline():
refs = []
for p in [1, 2, 3]:
ref = sub_pipeline(parameter=p)
refs.append(ref)

PipelineDecorator.wait_for_multi_pipelines()
# Aggregate results from all sub-pipelines

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					LonelyKangaroo55
				
					0
					 × 1

Write your answer

461 Views

2 Answers

5 months ago