Hi, I Am Running A Pipeline From Decorators. The Pipeline Runs Fine. Then I Try To Clone It By Clicking The (Successful) Run And Launching. The Pipeline Fails Immediately With The Error

Answered

hi, I am running a pipeline from decorators.
the pipeline runs fine.
Then I try to clone it by clicking the (successful) run and launching.

The pipeline fails immediately with the error
File "/usr/local/lib/python3.8/site-packages/clearml/automation/controller.py", line 3457, in internal_decorator a_pipeline._start(wait=False) File "/usr/local/lib/python3.8/site-packages/clearml/automation/controller.py", line 1213, in _start self._prepare_pipeline(step_task_completed_callback, step_task_created_callback) File "/usr/local/lib/python3.8/site-packages/clearml/automation/controller.py", line 1247, in _prepare_pipeline if not self._verify(): File "/usr/local/lib/python3.8/site-packages/clearml/automation/controller.py", line 1496, in _verify self._verify_node(node) File "/usr/local/lib/python3.8/site-packages/clearml/automation/controller.py", line 1512, in _verify_node raise ValueError("Node '{}', base_task_id is empty".format(node.name)) ValueError: Node 'my_task_1', base_task_id is emptyWhen I look at the json in the Pipeline configuration object I indeed see that for the (original, succeesful) run, the base_task_id fields are empty for all tasks.

How can I get it to clone properly?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					RoughTiger69
				
					0
					 × 1

Votes Newest

Answers 5

Hi RoughTiger69 , how are you running the pipeline? Locally or on agents? How is the controller running?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

CostlyOstrich36 all tasks are remote.
conrtoller - tried both

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					RoughTiger69
				
					0
					 × 1

CostlyOstrich36 I confirm this was the case.

So :

module_a.py @PipelineDecorator.pipeline()... from module_b import my_func x = my_func()
` modele_b.py

@PipelineDecorator.component()
def my_func()
pass `

Under this circumstances, the pipeline is created correctly and run correctly
But when I clone it (or click “Run” and submit) - it fails with the error above.

Moving my_func from module_a to module_b solves this.

To me this looks like a bug or unreasonable and undocumented limitation

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					RoughTiger69
				
					0
					 × 1

CostlyOstrich36 I’ve tried the pipeline_from_decorator.py example and it works.
Could it be a sensitivity to some components being on a different python .py file relative to the controller itself?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					RoughTiger69
				
					0
					 × 1

Any chance you have a toy script to play with?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

1K Views

5 Answers

2 years ago

one year ago