Hi. Looking Into Clearml Support For Datasets, I'D Like To Understand How To Work With Large Datasets And Cases Where Not All The Data Is Downloaded At Once. (E.G. 1. Each Training Epoch Is Performed On A (Preferably Random) Sample Of The Data That Is Dow

Answered

Hi.
Looking into clearml support for datasets, I'd like to understand how to work with large datasets and cases where not all the data is downloaded at once. (e.g. 1. Each training epoch is performed on a (preferably random) sample of the data that is downloaded at start of epoch and discarded at end of epoch or 2. Dataset is split such that each worker obtains part of the data)
I see that https://clear.ml/docs/latest/docs/references/sdk/dataset#get_local_copy has a part and num_parts that relates to chunks (which are determined and data upload time?).
Is that the mechanism I'm looking for?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					PanickyMoth78
				
					0
					 × 1

Votes Newest

Answers 3

Hi PanickyMoth78 , While the ClearML Datasets are meant to handle cases where the entire metadata fit in memory (or disk), the use-case you're describing is exactly where the HyperDatasets come into play, allowing you to use a backend-supported iterator(s) to (optionally randomly) iterate over your metadata (with automatic fetching and caching of raw data as required), which can also be used of course in cases where data split is required.

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

PanickyMoth78

Is it limited to

accounts? (

unfortunately, yes 😊 , but I'm sure sales will be able to hook you up ...

  				
Posted 
	one year ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

cool. How can I get started with hyper datasets? is it part of the clearml package?
Is it limited to https://clear.ml/pricing/?gclid=Cj0KCQjw5ZSWBhCVARIsALERCvzehkqVOiqJPaum5fsVyyTNMKce91PBHZd1IhQpEFaKvV7toze2A_0aAgXXEALw_wcB accounts?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					PanickyMoth78
				
					0
					 × 1

Write your answer

625 Views

3 Answers

one year ago