ClearML FAQ | What'S The Proper Way To Build A Pipeline?

Answered

What'S The Proper Way To Build A Pipeline?

What's the proper way to build a pipeline?
In ClearML there are several approaches to build a pipeline. I am about to make a pipeline for the following scenario.
1st task downloads a data and registers a dataset in ClearML. Result would be a dataset_id. 2nd task trains a model based on that dataset. Result would be a model. 3rd task takes trained model and converts it to tf light format. How can I pack all that tasks inside one pipeline nicely?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					GentleSwallow91
				
					0
					 × 1

Votes Newest

Answers

Hi GentleSwallow91
I think this would be a good start:
https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py
wdyt?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Write your answer

1K Views

1 Answer

2 years ago

one year ago

Tags

tf

Similar posts

2. I Have A Local Postgresql And Datafetcher Class, Whats The Best Way To Reuse Same Datafetcher In Local Runs With Pipeline. Is It Possible?

Hi, I Want To Build A Server In The Component Of Pipeline And Send Request To This Server From Another Component In Same Pipeline. I Get Static Ip From Dag But I Can'T To Do What I Want. I Need A Dynamic Ip That Is Shown On The Workers And Queues Page. Wh

Hey There!I’M Trying To Build A Pipeline With The Help Of Decorators. I Have A One-Step Pipeline, And The Step Is Imported From A Different Pipeline, Marked Up With The Component Decorator.

Hi, Quick Y/N Question: Is It Possible To Build A Pipeline Of Pipelines? I'D Imagine This To Be Possible, Since Pipelines Are Also Treated As Tasks.

Hi! I Am Trying To Build And Run A Pipeline. I Pass My Dataset As Parameter Of Pipeline:

Hi All! I Have A Question - I Have A Pipeline Build Out Of Multiple Tasks. One Of The Tasks Creates A File As An Output. I Don'T Need To Save This File Anywhere (Not In My Local Nor As An Artifact), But I Do Want To Use It An An Input For The Next Step In

Hello, I Am Currently Learning How To Build Pipelines With Clearml. I'Ve Created A Pipeline That Has Five Steps, In Which Each Step Depends On The Previous One (Step 2 Depends On Step 1, Step 3 Depends On Step 2, Etc.). However, My Step 4 Depends On Step

Hi, Question About Setting Output_Uri=True For A Pipeline Step Using Decorators. What I’M Seeing Is That Training Results In No Output Destination Specified, And Logs To Local Disk (When Run Remotely In Docker Container), Where Is The Proper Place To Set

Hi Everyone! I Have A Question About The Pipeline Controller: I Would Like To Build A Ml Pipeline Similar To The One At