ClearML FAQ | Hi, I Have A S3 Bucket Configured As Default Uri, But, When I Am Testing The Code On My Local Machine, I Do Not Want The Model Files To Be Uploaded To S3. I Tried Setting

Answered

Hi, I Have A S3 Bucket Configured As Default Uri, But, When I Am Testing The Code On My Local Machine, I Do Not Want The Model Files To Be Uploaded To S3. I Tried Setting

Hi, I have a s3 bucket configured as default uri, but, when I am testing the code on my local machine, I do not want the model files to be uploaded to s3. I tried setting output_uri="" but the files are still uploaded. If a specify a local folder, like mnt/Data , I guess the files will be copied to this folder. Is there a way to avoid uploading artifacts in a task, just using the original location of the artifacts in disk?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SkinnyPanda43
				
					0
					 × 1

Votes Newest

Answers 2

Hi SkinnyPanda43
In your local machine do not pass output_uri at all, so nothing will be uploaded.
On the agent's configuration file configure, default_output_uri to the S3 bucket
(Notice you can always override them in the UI, see the bottom of the execution Tab)
https://github.com/allegroai/clearml-agent/blob/e93384b99bdfd72a54cf2b68b3991b145b504b79/docs/clearml.conf#L312

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Thanks, I thought on doing that, but I was wondering if there was a way to set it into the code.

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SkinnyPanda43
				
					0
					 × 1

Write your answer

579 Views

2 Answers

2 years ago

one year ago

Tags

Similar posts

Does Dataset.Add_Files Support Uploading From S3 Uri? I Have No Problem Uploading To S3 But Cant Use Data That Is Already In S3? Or Am I Dong Something Wrong? I Read In Documentation That Add_External_Files Supports This Feature, But I Want To Be Able To

Hello. In Which Yaml Files For Kubernetes And Where Should I Specify The Aws File_Server And Credentials For Clearml-Agent So That When Running Tasks Remotely In The Queue, The Artifacts Are Saved In The S3 Bucket?

Hi Everyone, Does Clearml Preview *.Txt Files? I Uploaded About 2Gb Files And Want To Share It To Others And Make Sure They Can Preview Them. Please Help!

Hi Everyone, Weird Problem With Dataset.Get_Local_Copy (Both From Sdk And From Clearml-Data): I Have A Dataset With A Single File And Lots Of S3 Links. Used To Work Perfectly Until Those Files Started Becoming Larger (Or It Is Just A Matter Of Bad Timing

Hello Everyone, I Am Using Self Hosted Clearml Server On Ec2 (Clearml Community Amis). This Ec2 Instance Is Attached To S3 With Iam Role. Now If I Create Or Upload Data From Client Side , I Want It To Be Uploaded On S3. There Is A Way Mentioned For Mentio

I Wonder If There Is Any Way To Sync Data On Cloud Storage (E.G S3 Or Minio) Using Clearml-Data Sync Command? Because I Don’T See Any Place In The Documentation Mention It, We Can Upload From Local Files But Not The Files On Cloud

Hi Everyone! I'M Using Minios3 As A File Server And As Default Output Uri. I'Ve Faced The Following Problem. When I Delete Tasks From Web Ui (And Also From Archive) Their Artifacts Didn'T Get Deleted From S3. I'M Using Self Hosted Clearml==1.11. What Shou

Hi Guys! Is There Any Way To Get Full State Of Dataset From Somewhere, Except My S3 Bucket? I Need A Mapping Of Files And Batches That Were Uploaded As A Dataset. Maybe This Information Is Also Available In One Of The Clearml Databases?

[Pipeline] Hey, Is It Possible To Specify The Output Uri For Pipelines And Their Components Using Pipeline Decorators? I Would Like To Store Pipeline Artifacts And Component Artifacts On S3.

Hi Everyone, I’M Getting An Error During Model Upload To S3. The Error Shows Up In The Console Like Below And I Don’T See Any Uploaded Objects In S3: