Unanswered
Hi, I Am Using Aws Autoscalaer To Train Model. I Have A Fair Large Dataset(400G) And The Data Is Private So I Can'T Really Store It In Clearml Dataset. Everytime When I Launch A Job, It'S Going To Take Very Long Time To Download The Data From S3. Is There
Hi @<1597762318140182528:profile|EnchantingPenguin77> , in theory you could use EFS, but as far as I know the performance will not be very good, and using an AWS-native object storage (i.e. S3) in the same region should provide better performance.
An EBS volume can only be mapped to a single machine at a time, so when using the autoscaler, this can't be done (since a single volume can't be used for all machines the autoscaler spins)
146 Views
0
Answers
one year ago
one year ago