Hi, I Am Using Aws Autoscalaer To Train Model. I Have A Fair Large Dataset(400G) And The Data Is Private So I Can'T Really Store It In Clearml Dataset. Everytime When I Launch A Job, It'S Going To Take Very Long Time To Download The Data From S3. Is There

Unanswered

Hi @<1597762318140182528:profile|EnchantingPenguin77> , in theory you could use EFS, but as far as I know the performance will not be very good, and using an AWS-native object storage (i.e. S3) in the same region should provide better performance.
An EBS volume can only be mapped to a single machine at a time, so when using the autoscaler, this can't be done (since a single volume can't be used for all machines the autoscaler spins)

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

65 Views

0 Answers

7 months ago