Hi Quick Question About Clearml-Data: It Seems Like It Is Only Using A Single Thread For Compression. Is There A Way To Use Multi-Threading? For Big Datasets It Takes Quite A While To Upload.

Answered

Hi quick question about clearml-data: It seems like it is only using a single thread for compression. Is there a way to use multi-threading? For big datasets it takes quite a while to upload.

  				
Posted 
	one year ago

					More  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

Votes Newest

Answers 3

Hi ReassuredTiger98 , how are you currently uploading? You can use the max_workers parameter to use multiple threads

  				
Posted 
	one year ago

					More  		
  Report
		
					CostlyOstrich36
				
					0

Hi CostlyOstrich36
Thanks a lot. It seems there is no such option in clearml-data CLI, right?

  				
Posted 
	one year ago

					More  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

Also, is max_workers about compression threads or upload threads or both?

  				
Posted 
	one year ago

					More  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

Write your answer

902 Views

3 Answers

one year ago