Hi all, quick question about ClearML Datasets. Does anyone know if it is possible to access (could just be paths to the data in a bucket) a dataset directly ...
7 months ago
AmiableSheep6
1
Question,
2
Answers
Active since 05 April 2024
Last activity
7 months ago
Reputation
0
Badges 1
2 × Eureka!
0
Hi All, Quick Question About Clearml Datasets.
Does Anyone Know If It Is Possible To Access (Could Just Be Paths To The Data In A Bucket) A Dataset Directly From S3, Instead Of Downloading A Local Copy? We Typically Store And Access Large Quantities Of
Or even better, get a list of the paths to the data locations on S3 (captured within a tracked dataset) which I can pull through boto3 etc
7 months ago
0
Hi All, Quick Question About Clearml Datasets.
Does Anyone Know If It Is Possible To Access (Could Just Be Paths To The Data In A Bucket) A Dataset Directly From S3, Instead Of Downloading A Local Copy? We Typically Store And Access Large Quantities Of
Thank you for the quick response! StorageManager might help… I think what I actually need is a way to get the data in mini batches, instead of caching the entire dataset with get_local_copy(). Bc, yes, I will have to download the data anyway for training
7 months ago