ClearML FAQ | Hi All! I Have A Question - I Have A Pipeline Build Out Of Multiple Tasks. One Of The Tasks Creates A File As An Output. I Don'T Need To Save This File Anywhere (Not In My Local Nor As An Artifact), But I Do Want To Use It An An Input For The Next Step In

Answered

Hi All! I Have A Question - I Have A Pipeline Build Out Of Multiple Tasks. One Of The Tasks Creates A File As An Output. I Don'T Need To Save This File Anywhere (Not In My Local Nor As An Artifact), But I Do Want To Use It An An Input For The Next Step In

Hi all!
I have a question - I have a pipeline build out of multiple tasks. One of the tasks creates a file as an output. I don't need to save this file anywhere (not in my local nor as an artifact), but I do want to use it an an input for the next step in the pipeline.
Is there a way to use this output without uploading this file as artifact?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					DangerousMole43
				
					0
					 × 1

Votes Newest

Answers 4

Hi @<1539417873305309184:profile|DangerousMole43> , unless both steps run on the same machine you will have to upload it as an artifact somehow

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Hi @<1539417873305309184:profile|DangerousMole43> , in that case I think you can simply save the file path as a configuration in the first step and then in the next step you can simply access this file path from the previous step. Makes sense?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

yes!
I'll try that out. Thank you @<1523701070390366208:profile|CostlyOstrich36>

  				
Posted 
	one year ago

					More
				  		
  Report
		
					DangerousMole43
				
					0
					 × 1

Thanks @<1523701070390366208:profile|CostlyOstrich36> !
so, both steps are running on the same machine. I would love to get your suggestion in this case.

  				
Posted 
	one year ago

					More
				  		
  Report
		
					DangerousMole43
				
					0
					 × 1

Write your answer

731 Views

4 Answers

one year ago

one year ago

Tags

Similar posts

Hi Everyone! I'M Using Minios3 As A File Server And As Default Output Uri. I'Ve Faced The Following Problem. When I Delete Tasks From Web Ui (And Also From Archive) Their Artifacts Didn'T Get Deleted From S3. I'M Using Self Hosted Clearml==1.11. What Shou

Hi Guys, I'M Setting Up A Bunch Of Machines As Clearml Agents And Have Run Into An Issue With Caching. We Are Using Poetry For Python Dependency Management, So The Agents Are Configured To Use That Too, But They Are Not Caching The Venvs Between Tasks. Th

Hi, I Was Trying Out The Clearml Pipelines From Task. I Had 2 Tasks That Have Run Successfully As Clearml Task. One Of The Tasks Has Dependency On External Python File. I.E. I Have Used From <External File> Import Methoda In My First Task. It Executed Fin

Hi, I Just Started Using Clearml, And It Is Amazing! However I'Ve Run Into An Issue - I Have A Windows Machine Which I'Ve Setup As A Worker, With An Agent Running. I'Ve Thus Far Been Able To Run The Hello World Tasks And Have Also Set It Up To Store All D

Hi, I Am Using Minio To Save My Files. And Recently I Have Encountered An Issue With Downloading The Artifacts, With Compressed Csv Files (Test_Prediction_Df.Csv.Gz) In Particular. When I Try To Download These Files From The Web Ui, The File Opens As A Pl

Is There An Easy Way To Add A Link To One Of The Tasks Panels? (As An Artifact, Configuration, Info, Etc)? Edit: And Follow Up Regarding The Dataset. As Discussed Somewhere Previously, The Datasets Are Now Automatically Moved To A Hidden "Sub-Project" Pr

Hi All, I'M Running Some Training Tasks Remotely Using Clearml Agent, Using My Github Repo Code. One Of The Input Arguments To My Process Is A Config Yaml File. The Pipeline Is Setup So That This Config File Is Referenced With A Relative Path Such As

Hey, So I'M Trying To Upload An Artefact To Clearml’S Fileserver(I Have A Self Hosted Clearml Server Running), I'Ve Uploaded The File Using Storagemanager.Upload_File(Path, Url) And Giving The Url As “

We Have A Environment Variables Definitions.Py File Which Every User Configures On Their Local Machine. This File Includes Local Paths As Well As Aws/Api Credentials. This Is An Issue When Spinning Up Clearml Tasks Since It Is Not Included In The Git Repo

Hey Guys, I'M Hosting A Private Server Configured To Link To A S3 Bucket. I'M Having Difficulties Identifying The Reason For An Error In A Worker Recurring Repeatedly (Shown In The Screenshots Attached). It Basically Uses The Same Clearml.Conf File As The