ClearML FAQ | Hi All! I Have A Question - I Have A Pipeline Build Out Of Multiple Tasks. One Of The Tasks Creates A File As An Output. I Don'T Need To Save This File Anywhere (Not In My Local Nor As An Artifact), But I Do Want To Use It An An Input For The Next Step In

Answered

Hi All! I Have A Question - I Have A Pipeline Build Out Of Multiple Tasks. One Of The Tasks Creates A File As An Output. I Don'T Need To Save This File Anywhere (Not In My Local Nor As An Artifact), But I Do Want To Use It An An Input For The Next Step In

Hi all!
I have a question - I have a pipeline build out of multiple tasks. One of the tasks creates a file as an output. I don't need to save this file anywhere (not in my local nor as an artifact), but I do want to use it an an input for the next step in the pipeline.
Is there a way to use this output without uploading this file as artifact?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					DangerousMole43
				
					0
					 × 1

Votes Newest

Answers 4

Thanks @<1523701070390366208:profile|CostlyOstrich36> !
so, both steps are running on the same machine. I would love to get your suggestion in this case.

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					DangerousMole43
				
					0
					 × 1

Hi @<1539417873305309184:profile|DangerousMole43> , unless both steps run on the same machine you will have to upload it as an artifact somehow

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Hi @<1539417873305309184:profile|DangerousMole43> , in that case I think you can simply save the file path as a configuration in the first step and then in the next step you can simply access this file path from the previous step. Makes sense?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

yes!
I'll try that out. Thank you @<1523701070390366208:profile|CostlyOstrich36>

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					DangerousMole43
				
					0
					 × 1

Write your answer

1K Views

4 Answers

2 years ago

2 years ago

Tags

Similar posts

Hi, I Tried To Build My Own Image And Uploaded It To Harbor In Order To Use It As An Image For Pipelines And Tasks, But The Tasks Didn'T Use The Packages From It. Dockefile: From Nvidia/Cuda:12.4.0-Devel-Ubuntu22.04 Env Tz=Europe/Moscow Run Ln -Snf /Usr

As Soon As I Refactor My Project Into Multiple Folders, Where On Top-Level I Put My Pipeline File, And Keep My Tasks In A Subfolder, The Clearml Agent Seems To Have Problems:

Hi All, My Name Is Evgeny From Corsound. I Have A Question. I Have A Dataset With Audio Files, In The Preview Bar I Can Select A File And Play It Inside An Iframe As It Has A Url. Now I'M Also Using Twilio, It Can Get A Url And Play It As A Recording. Is

Hey All, I Have A Question That Might Have Been Asked Before But I Can'T Find An Answer On This Slack Channel Or The Github So I Hope Somebody Can Help Me. I Have Created A Dataset Via The Python Sdk With A Local Path As The Output Uri Like So:

Hi Everyone! I'M Using Minios3 As A File Server And As Default Output Uri. I'Ve Faced The Following Problem. When I Delete Tasks From Web Ui (And Also From Archive) Their Artifacts Didn'T Get Deleted From S3. I'M Using Self Hosted Clearml==1.11. What Shou

Hey Guys! I Was Wondering If There Are Any Good Examples In Deployment Of Cv (Detection) Models Through Clearml-Serving? The Examples I Have Seen So Far Are Classification, And So The Post-Processing From The Triton Output Is Minimal. As An Example Mayb

Hi Guys, I'M Setting Up A Bunch Of Machines As Clearml Agents And Have Run Into An Issue With Caching. We Are Using Poetry For Python Dependency Management, So The Agents Are Configured To Use That Too, But They Are Not Caching The Venvs Between Tasks. Th

Hi, I Was Trying Out The Clearml Pipelines From Task. I Had 2 Tasks That Have Run Successfully As Clearml Task. One Of The Tasks Has Dependency On External Python File. I.E. I Have Used From <External File> Import Methoda In My First Task. It Executed Fin

Hi, I Just Started Using Clearml, And It Is Amazing! However I'Ve Run Into An Issue - I Have A Windows Machine Which I'Ve Setup As A Worker, With An Agent Running. I'Ve Thus Far Been Able To Run The Hello World Tasks And Have Also Set It Up To Store All D

Hi, I Am Using Minio To Save My Files. And Recently I Have Encountered An Issue With Downloading The Artifacts, With Compressed Csv Files (Test_Prediction_Df.Csv.Gz) In Particular. When I Try To Download These Files From The Web Ui, The File Opens As A Pl