ClearML FAQ | Hello, I Have A General Question About Data Versioning Using Clearml. When Lets Say That My Parent Dataset Has 100 Files, And That I Create A Child Dataset From It By Adding An Extra 50 Files To The Original 100. Will My 100 Files Be Duplicated On My Serv

Answered

Hello, I Have A General Question About Data Versioning Using Clearml. When Lets Say That My Parent Dataset Has 100 Files, And That I Create A Child Dataset From It By Adding An Extra 50 Files To The Original 100. Will My 100 Files Be Duplicated On My Serv

hello, i have a general question about data versioning using ClearML.
When lets say that my parent dataset has 100 files, and that I create a child dataset from it by adding an extra 50 files to the original 100. Will my 100 files be duplicated on my server?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					MassiveGoldfish6
				
					0
					 × 1

Votes Newest

Answers 5

is this still true if the child dataset is smaller than the parent? If the parent dataset is 1Tb and I delete half the files, I will still be pushing 2Tb of data to the server?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					MassiveGoldfish6
				
					0
					 × 1

No, it should be just the amount of files remaining

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Hi @<1547028031053238272:profile|MassiveGoldfish6> , yes, every new version contains all included files

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Yes. Differential Datasets are part of the ClearML Scale and Enterprise solution 😞

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

so if my parent dataset is 1Tb and I add a single file to create a child dataset. There will now be 2Tb of data on the server. The parent dataset is duplicated on the server?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					MassiveGoldfish6
				
					0
					 × 1

Write your answer

1K Views

5 Answers

one year ago

one year ago

Tags

Similar posts

Hi All, My Name Is Evgeny From Corsound. I Have A Question. I Have A Dataset With Audio Files, In The Preview Bar I Can Select A File And Play It Inside An Iframe As It Has A Url. Now I'M Also Using Twilio, It Can Get A Url And Play It As A Recording. Is

Hi, I Have Recently Found Out That Storagemanager Can Be Used On Its Own And Since Then I Have Been Using It Together With Clearml'S Fileserver To Store Some Files That Are Not Related To A Particular Experiment Or Dataset. My Question Is, How Can I Dele

Hi Guys! Is There Any Way To Get Full State Of Dataset From Somewhere, Except My S3 Bucket? I Need A Mapping Of Files And Batches That Were Uploaded As A Dataset. Maybe This Information Is Also Available In One Of The Clearml Databases?

In My Current Project I Generate The Data From An Sql Query. Is The Only Way To Register The Dataset With Clearml To Write The Files To Disk First Or Is There Another Method? This Leads Into The Second Issue I Have, Which Is What Happens When I Store The

Ps C:\Xxxx> Clearml-Data Sync --Name Gbsg --Project Verify_Ds --Folder C:\Xxxx --P Arents Xxxx Creating A New Dataset: New Dataset Created Id=Xxxx Syncing Dataset Id Xxxx To Local Folder C:\Xxxx Generating Sha2 Hash For 2 Files 100%|██████████████████████

Hi, Newbe Question In Clearml Data Assuming My Files Are Already In A Bucket On The Cloud - What Would Be The Best Practice To Create A Dataset And Register It In Clearml?

I Get This Error When I Try To Add A Batch To A In My Clearml-Dataset. Generating Sha2 Hash For 100 Files Hash Generation Completed 2021-09-10 10:49:13,042 - Clearml.Task - Error - Action Failed <400/110: Tasks.Add_Or_Update_Artifacts/V2.10 (Invalid Task

Hi, I'M Trying To Understand If Clearml Supports My Usecase: I Generate My Own Data (Problem-Specific) And Generation Process Is Uploading A Parquet Dataset (In Fact, Parquet Contains Multiple Files Obv.) To S3. Is There A Way To "Register" The Dataset In