ClearML FAQ | Hi, I Have A S3 Bucket Configured As Default Uri, But, When I Am Testing The Code On My Local Machine, I Do Not Want The Model Files To Be Uploaded To S3. I Tried Setting

Answered

Hi, I Have A S3 Bucket Configured As Default Uri, But, When I Am Testing The Code On My Local Machine, I Do Not Want The Model Files To Be Uploaded To S3. I Tried Setting

Hi, I have a s3 bucket configured as default uri, but, when I am testing the code on my local machine, I do not want the model files to be uploaded to s3. I tried setting output_uri="" but the files are still uploaded. If a specify a local folder, like mnt/Data , I guess the files will be copied to this folder. Is there a way to avoid uploading artifacts in a task, just using the original location of the artifacts in disk?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					SkinnyPanda43
				
					0
					 × 1

Votes Newest

Answers 2

Hi SkinnyPanda43
In your local machine do not pass output_uri at all, so nothing will be uploaded.
On the agent's configuration file configure, default_output_uri to the S3 bucket
(Notice you can always override them in the UI, see the bottom of the execution Tab)
https://github.com/allegroai/clearml-agent/blob/e93384b99bdfd72a54cf2b68b3991b145b504b79/docs/clearml.conf#L312

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Thanks, I thought on doing that, but I was wondering if there was a way to set it into the code.

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					SkinnyPanda43
				
					0
					 × 1

Write your answer

973 Views

2 Answers

3 years ago

one year ago

Tags

Similar posts

Hi, I Have A Question About The Model Registry. Here'S My Situation: I'M Using K8S_Example And Struggling With Uploading A Model. Should Models Be Uploaded To The Fileserver, Or Should I Create Another S3 Bucket As Mentioned In The Documentation?

Is This A Bug? Clearml Version 1.26.4 Im Passing A List Of S3 Urls To Add_External_Files The List Is About 1K Of Files, But Only 7 Are Uploaded (Only Unique File Basenames) Here Is An Example: External_Files = [ '

I Have A Dataset Uploaded To Clearml. By Mistake I Didn'T Set The Destination Correctly And The Dataset Is Now Saved On A Computer Rather Than In S3. Is There An Easy Way To Rectify This?

Hi I Am Creating A Custom Image Dataset And Does Anyone Know How To Upload Files Into Remote Uri (S3) Without Compressing The Data?

Does Dataset.Add_Files Support Uploading From S3 Uri? I Have No Problem Uploading To S3 But Cant Use Data That Is Already In S3? Or Am I Dong Something Wrong? I Read In Documentation That Add_External_Files Supports This Feature, But I Want To Be Able To

Hello. In Which Yaml Files For Kubernetes And Where Should I Specify The Aws File_Server And Credentials For Clearml-Agent So That When Running Tasks Remotely In The Queue, The Artifacts Are Saved In The S3 Bucket?

Hi Everyone, Does Clearml Preview *.Txt Files? I Uploaded About 2Gb Files And Want To Share It To Others And Make Sure They Can Preview Them. Please Help!

Hi Everyone, Weird Problem With Dataset.Get_Local_Copy (Both From Sdk And From Clearml-Data): I Have A Dataset With A Single File And Lots Of S3 Links. Used To Work Perfectly Until Those Files Started Becoming Larger (Or It Is Just A Matter Of Bad Timing

Hello Everyone, I Am Using Self Hosted Clearml Server On Ec2 (Clearml Community Amis). This Ec2 Instance Is Attached To S3 With Iam Role. Now If I Create Or Upload Data From Client Side , I Want It To Be Uploaded On S3. There Is A Way Mentioned For Mentio

I Wonder If There Is Any Way To Sync Data On Cloud Storage (E.G S3 Or Minio) Using Clearml-Data Sync Command? Because I Don’T See Any Place In The Documentation Mention It, We Can Upload From Local Files But Not The Files On Cloud