Hi Everyone! I Have A Clearml Dataset That Takes Up 10 Tb. Its Local Download (Get_Local_Copy) Takes About A Month. Can You Tell Me How To Speed Up This Process? I Keep The Cache In Nfs.

Answered

Hi everyone!
I have a ClearML dataset that takes up 10 Tb. Its local download (get_local_copy) takes about a month. Can you tell me how to speed up this process? I keep the cache in nfs.

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					GrievingKoala83
				
					0
					 × 1

Votes Newest

Answers 2

Hi @<1578555761724755968:profile|GrievingKoala83> , with such huge files, I would think storing the files in NAS to begin with (using the file:// scheme, no uploading to the fileserver) should make this manageable

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

okey, I see, thank you!

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					GrievingKoala83
				
					0
					 × 1

Write your answer

474 Views

2 Answers

5 months ago