Is There A Way To Do S3 -> S3 Copy While Doing A Dataset? I Don’T Want To Get It To Local From S3 And Then Upload As A Dataset To S3

Answered

Is there a way to do s3 -> s3 copy while doing a dataset? I don’t want to get it to local from s3 and then upload as a dataset to s3

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					TrickySheep9
				
					0
					 × 1

Votes Newest

Answers 4

When uploading the files, hash is being calculated for every entry, and this is done for the local files. so currently clearml-data support local files.

What would you like to do with the dataset? Why not using it directly from S3?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					TimelyPenguin76
				
					0
					 Administrator

Hi TrickySheep9 , didnt get the idea, you are using clearml-data ? you just want to upload a local folder to S3?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					TimelyPenguin76
				
					0
					 Administrator

Yes using clearml-data.

Can I pass a s3 path to ds.add_files() essentially so that I can directly store a dataset without having to get the files to local and then upload again. Makes sense?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					TrickySheep9
				
					0
					 × 1

Why not using it directly from S3?

You can https://allegro.ai/clearml/docs/docs/examples/examples_storagehelper.html#downloading-a-file it with the storageManager

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					TimelyPenguin76
				
					0
					 Administrator

Write your answer

1K Views

4 Answers

3 years ago

one year ago

Is There Any Way To Get Dataset Size Without Downloading State.Json? Im Doing Ds = Clearml.Dataset.Get(Dataset_Id=D_Id), But It Instantly Tries To Download State.Json Which Is On S3. Im Only Interested In Size And File Count Which I Then Get From Calling

I Have A Dataset Uploaded To Clearml. By Mistake I Didn'T Set The Destination Correctly And The Dataset Is Now Saved On A Computer Rather Than In S3. Is There An Easy Way To Rectify This?

Hi I Am Creating A Custom Image Dataset And Does Anyone Know How To Upload Files Into Remote Uri (S3) Without Compressing The Data?

Hi All, Quick Question About Clearml Datasets. Does Anyone Know If It Is Possible To Access (Could Just Be Paths To The Data In A Bucket) A Dataset Directly From S3, Instead Of Downloading A Local Copy? We Typically Store And Access Large Quantities Of

Hello Everybody, I Have A Quick Question, I Have Been Training Multiple Models And Have Reached A Point Where I Am Satisfied With A Model. How Do I Push My Model To S3 Server( Setup In The Clearml.Conf File)? Similar To What We Do When Pushing A Dataset

Hi Everyone, A Question About Clearml-Data: Is It Possible To Tell Clearml To Not Chunk And Zip My Data, But Instead Rebuild Keep The Structure Of The Dataset? E.G. If My Data Is Image1.Jpg, Image2.Jpy, ... I Want To See The Same On S3. This Would Enable

Hi Everyone, Weird Problem With Dataset.Get_Local_Copy (Both From Sdk And From Clearml-Data): I Have A Dataset With A Single File And Lots Of S3 Links. Used To Work Perfectly Until Those Files Started Becoming Larger (Or It Is Just A Matter Of Bad Timing

Hi, I Am Using Aws Autoscalaer To Train Model. I Have A Fair Large Dataset(400G) And The Data Is Private So I Can'T Really Store It In Clearml Dataset. Everytime When I Launch A Job, It'S Going To Take Very Long Time To Download The Data From S3. Is There

Hi. Can You Point Me In A Direction Of The Solution Please? We Are Trying To Use Spot Machines On Aws Through Autoscaler In Clearml. The Problem Is We Need To Download Each Time Huge Dataset From S3. Is There Some Cash Solution, Like Attaching Disk For Ex