cool. How can I get started with hyper datasets? is it part of the clearml package?
Is it limited to https://clear.ml/pricing/?gclid=Cj0KCQjw5ZSWBhCVARIsALERCvzehkqVOiqJPaum5fsVyyTNMKce91PBHZd1IhQpEFaKvV7toze2A_0aAgXXEALw_wcB accounts?
PanickyMoth78
Is it limited to
accounts? (
unfortunately, yes 😊 , but I'm sure sales will be able to hook you up ...
Hi PanickyMoth78 , While the ClearML Datasets are meant to handle cases where the entire metadata fit in memory (or disk), the use-case you're describing is exactly where the HyperDatasets come into play, allowing you to use a backend-supported iterator(s) to (optionally randomly) iterate over your metadata (with automatic fetching and caching of raw data as required), which can also be used of course in cases where data split is required.