Hi Guys, I Am Trying To Use Clearml Pipelines By Function. May I Know The Correct Way To Pass Model/Files From One Tasks To Next? I Realised Pickle May Not Work Sometimes. Also, Some Steps May Require Complete Set Of Files Before It Should Starts, Is Th

Answered

Hi guys, I am trying to use clearml pipelines by function. May i know the correct way to pass model/files from one tasks to next? I realised pickle may not work sometimes.

Also, some steps may require complete set of files before it should starts, is there a way to define this dependency so the step only starts when it should.

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					OddShrimp85
				
					0
					 × 1

Votes Newest

Answers 4

Thanks. The examples uses upload_artifact which stores the files in output_uri. What if I do not want to save it but simply pass to next step, is there a way to do so?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					OddShrimp85
				
					0
					 × 1

Hi @<1523701304709353472:profile|OddShrimp85> , I would suggest looking at the examples here:
None

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Pipelines assume that different steps run on different machines. How would you pass those files between the different machines? If the steps run on the same machine then why have them as different steps?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Another issue that I am facing is I am unable to call different functions from other modules present in the same directory.

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					ArrogantButterfly10
				
					0
					 × 1

Write your answer

594 Views

4 Answers

10 months ago