Is There A Way To Do S3 -> S3 Copy While Doing A Dataset? I Don’T Want To Get It To Local From S3 And Then Upload As A Dataset To S3

Answered

Is there a way to do s3 -> s3 copy while doing a dataset? I don’t want to get it to local from s3 and then upload as a dataset to s3

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TrickySheep9
				
					0
					 × 1

Votes Newest

Answers 4

Why not using it directly from S3?

You can https://allegro.ai/clearml/docs/docs/examples/examples_storagehelper.html#downloading-a-file it with the storageManager

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TimelyPenguin76
				
					0
					 Administrator

Yes using clearml-data.

Can I pass a s3 path to ds.add_files() essentially so that I can directly store a dataset without having to get the files to local and then upload again. Makes sense?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TrickySheep9
				
					0
					 × 1

When uploading the files, hash is being calculated for every entry, and this is done for the local files. so currently clearml-data support local files.

What would you like to do with the dataset? Why not using it directly from S3?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TimelyPenguin76
				
					0
					 Administrator

Hi TrickySheep9 , didnt get the idea, you are using clearml-data ? you just want to upload a local folder to S3?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TimelyPenguin76
				
					0
					 Administrator

Write your answer

710 Views

4 Answers

2 years ago

one year ago

Hello Everybody, I Have A Quick Question, I Have Been Training Multiple Models And Have Reached A Point Where I Am Satisfied With A Model. How Do I Push My Model To S3 Server( Setup In The Clearml.Conf File)? Similar To What We Do When Pushing A Dataset

Hi Everyone, A Question About Clearml-Data: Is It Possible To Tell Clearml To Not Chunk And Zip My Data, But Instead Rebuild Keep The Structure Of The Dataset? E.G. If My Data Is Image1.Jpg, Image2.Jpy, ... I Want To See The Same On S3. This Would Enable

Hi Everyone, Weird Problem With Dataset.Get_Local_Copy (Both From Sdk And From Clearml-Data): I Have A Dataset With A Single File And Lots Of S3 Links. Used To Work Perfectly Until Those Files Started Becoming Larger (Or It Is Just A Matter Of Bad Timing

Hi, I Am Using Aws Autoscalaer To Train Model. I Have A Fair Large Dataset(400G) And The Data Is Private So I Can'T Really Store It In Clearml Dataset. Everytime When I Launch A Job, It'S Going To Take Very Long Time To Download The Data From S3. Is There

Hi. Can You Point Me In A Direction Of The Solution Please? We Are Trying To Use Spot Machines On Aws Through Autoscaler In Clearml. The Problem Is We Need To Download Each Time Huge Dataset From S3. Is There Some Cash Solution, Like Attaching Disk For Ex

Hi! How Can I Delete Dataset From Ui And From S3 Bucket? I Tried To Delete From Ui And Then Checking But I Still Have It ... Api Client Doesn'T Have Methods To Work With Datasets ...

Hi All, I Dont Know What Happened But I Am Unable To Download A Dataset I Used To Download To Cached Folder. Now, When I Try To Download, The Dataset Show The Following Error. Just Few Day Ago, I Still Can Download And Run With The Dataset Sucessfully. I

Hi Guys! Is There Any Way To Get Full State Of Dataset From Somewhere, Except My S3 Bucket? I Need A Mapping Of Files And Batches That Were Uploaded As A Dataset. Maybe This Information Is Also Available In One Of The Clearml Databases?

Is There A Way I Can Upload An Ml Model (Raw, No Training Task) To S3 And Log It In The Models Section Of Clearml. I Succeeded With Storage Manager But Dont Know How To Make It Visible In Models Section. Also My Model Is Sometimes A Single File And Someti