Unanswered
Does Dataset.Add_Files Support Uploading From S3 Uri?
I Have No Problem Uploading To S3 But Cant Use Data That Is Already In S3?
Or Am I Dong Something Wrong? I Read In Documentation That Add_External_Files Supports This Feature, But I Want To Be Able To
Our datasets are more than 1TB in size and will grow in size (probably 4TB and up), this means we also need 4TB local storage just to upload the dataset back in zipped format. This is not a good solution.
What we can do I guess is do the downloading locally by some chunks of files?
Download locally 100 files, add_to_clearml dataset, repeat
86 Views
0
Answers
7 months ago
7 months ago