ClearML FAQ | Hi All! I Have A Question - I Have A Pipeline Build Out Of Multiple Tasks. One Of The Tasks Creates A File As An Output. I Don'T Need To Save This File Anywhere (Not In My Local Nor As An Artifact), But I Do Want To Use It An An Input For The Next Step In

Answered

Hi All! I Have A Question - I Have A Pipeline Build Out Of Multiple Tasks. One Of The Tasks Creates A File As An Output. I Don'T Need To Save This File Anywhere (Not In My Local Nor As An Artifact), But I Do Want To Use It An An Input For The Next Step In

Hi all!
I have a question - I have a pipeline build out of multiple tasks. One of the tasks creates a file as an output. I don't need to save this file anywhere (not in my local nor as an artifact), but I do want to use it an an input for the next step in the pipeline.
Is there a way to use this output without uploading this file as artifact?

  				
Posted 
	2 years ago

					More  		
  Report
		
					DangerousMole43
				
					0
					 × 1

Votes Newest

Answers 4

Hi DangerousMole43 , unless both steps run on the same machine you will have to upload it as an artifact somehow

  				
Posted 
	2 years ago

					More  		
  Report
		
					CostlyOstrich36
				
					0

Hi DangerousMole43 , in that case I think you can simply save the file path as a configuration in the first step and then in the next step you can simply access this file path from the previous step. Makes sense?

  				
Posted 
	2 years ago

					More  		
  Report
		
					CostlyOstrich36
				
					0

yes!
I'll try that out. Thank you CostlyOstrich36

  				
Posted 
	2 years ago

					More  		
  Report
		
					DangerousMole43
				
					0
					 × 1

Thanks CostlyOstrich36 !
so, both steps are running on the same machine. I would love to get your suggestion in this case.

  				
Posted 
	2 years ago

					More  		
  Report
		
					DangerousMole43
				
					0
					 × 1

Write your answer

2K Views

4 Answers

2 years ago

2 years ago

Tags

Similar posts

Hi All, My Name Is Evgeny From Corsound. I Have A Question. I Have A Dataset With Audio Files, In The Preview Bar I Can Select A File And Play It Inside An Iframe As It Has A Url. Now I'M Also Using Twilio, It Can Get A Url And Play It As A Recording. Is

Hey All, I Have A Question That Might Have Been Asked Before But I Can'T Find An Answer On This Slack Channel Or The Github So I Hope Somebody Can Help Me. I Have Created A Dataset Via The Python Sdk With A Local Path As The Output Uri Like So:

Hi Everyone! I'M Using Minios3 As A File Server And As Default Output Uri. I'Ve Faced The Following Problem. When I Delete Tasks From Web Ui (And Also From Archive) Their Artifacts Didn'T Get Deleted From S3. I'M Using Self Hosted Clearml==1.11. What Shou

Hey Guys! I Was Wondering If There Are Any Good Examples In Deployment Of Cv (Detection) Models Through Clearml-Serving? The Examples I Have Seen So Far Are Classification, And So The Post-Processing From The Triton Output Is Minimal. As An Example Mayb

Hi, I Am Using Minio To Save My Files. And Recently I Have Encountered An Issue With Downloading The Artifacts, With Compressed Csv Files (Test_Prediction_Df.Csv.Gz) In Particular. When I Try To Download These Files From The Web Ui, The File Opens As A Pl

Hey, So I'M Trying To Upload An Artefact To Clearml’S Fileserver(I Have A Self Hosted Clearml Server Running), I'Ve Uploaded The File Using Storagemanager.Upload_File(Path, Url) And Giving The Url As “

Hi, I Am Trying To Save Task Artifacts On Gcs Bucket - I Have Configured The Project To Use The Bucket As “Default Output Destination” (In The Project Create Dialog) But The Artifacts Still Created Inside The File Server, Any Idea What Am I Missing?

We Have A Environment Variables Definitions.Py File Which Every User Configures On Their Local Machine. This File Includes Local Paths As Well As Aws/Api Credentials. This Is An Issue When Spinning Up Clearml Tasks Since It Is Not Included In The Git Repo

Hi, When I Save Model Using Tf.Keras.Save_Model Or Using Modelcheckpoint Model Is Not Saved As An Artifact. Output Uri Is Set To Google Cloud Bucket. When Reporting With Logger Everything Is Stored Correctly. Do You Maybe Have Any Idea Why This Would Not

Hey Guys, I'M Hosting A Private Server Configured To Link To A S3 Bucket. I'M Having Difficulties Identifying The Reason For An Error In A Worker Recurring Repeatedly (Shown In The Screenshots Attached). It Basically Uses The Same Clearml.Conf File As The