Hi, I'M Trying To Reproduce The Pipeline Example

Answered

Hi, I'm trying to reproduce the pipeline example https://github.com/allegroai/clearml/tree/master/examples/pipeline and execute it remotely. It results with this error ValueError: Could not find base_task_project=pipeline_examples base_task_name=pipeline step 1 dataset artifact . So I'm wondering how the pipeline is ultimately supposed to find the step. Generally is there any tutorial to execute the examples remotely in the clearml agents (either by python script or clearml-task cli)?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					SplendidToad10
				
					0

Votes Newest

Answers 4

Hi SplendidToad10
In order to run a pipeline you first have to create the steps (i.e Tasks).
This is usually dont by running the code once (basically running any code with Task.init call will create a Task for that specific code, including the enviroement definition needed to reproduce it by the Agent)

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Just run once (from your python console / pycharm etc.):
https://github.com/allegroai/clearml/blob/master/examples/automation/toy_base_task.py

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

https://allegro.ai/clearml/docs/docs/examples/pipeline/pipeline_controller.html

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

BTW: there is a full Pipeline class that does everything for you, example here:
https://github.com/allegroai/clearml/tree/master/examples/pipeline

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Write your answer

1K Views

4 Answers

3 years ago

2 years ago