Hi, I Am Creating Pipeline And Have Several Step/Task. Example Step1 Will Download Data By Varibel Range-Date. Is It Possible, If The Parameter Of Step1 Not Changing (In This Case Range-Date), It Will Not Re-Download And Not Re-Upload It To Artifacts, But

Answered

hi,
i am creating pipeline and have several step/task.
example step1 will download data by varibel range-date.
is it possible, if the parameter of step1 not changing (in this case range-date), it will not re-download and not re-upload it to artifacts, but can directly pointing the artifact with that variable?
TIA

  				
Posted 
	one year ago

					More  		
  Report
		
					QuaintJellyfish58
				
					0
					 × 1

Votes Newest

Answers 5

Hi QuaintJellyfish58 , I think what you're referring to is caching steps - if nothing changed code or configuration wise then use outputs from previous pipeline run. Is that what you're looking for?

  				
Posted 
	one year ago

					More  		
  Report
		
					CostlyOstrich36
				
					0

clearml is awsome ❤ 🤩

  				
Posted 
	one year ago

					More  		
  Report
		
					QuaintJellyfish58
				
					0
					 × 1

Hi, CostlyOstrich36 ,yes! correct! how to achive that? it will save my storage.

  				
Posted 
	one year ago

					More  		
  Report
		
					QuaintJellyfish58
				
					0
					 × 1

QuaintJellyfish58 , check cache_executed_step parameter in pipeline steps

  				
Posted 
	one year ago

					More  		
  Report
		
					CostlyOstrich36
				
					0

None

  				
Posted 
	one year ago

					More  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

1K Views

5 Answers

one year ago