Hi Quick Question About Clearml-Data: It Seems Like It Is Only Using A Single Thread For Compression. Is There A Way To Use Multi-Threading? For Big Datasets It Takes Quite A While To Upload.

Answered

Hi quick question about clearml-data: It seems like it is only using a single thread for compression. Is there a way to use multi-threading? For big datasets it takes quite a while to upload.

  				
Posted 
	8 months ago

					More
				  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

Votes Newest

Answers 3

Hi @<1523701868901961728:profile|ReassuredTiger98> , how are you currently uploading? You can use the max_workers parameter to use multiple threads

  				
Posted 
	8 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Also, is max_workers about compression threads or upload threads or both?

  				
Posted 
	8 months ago

					More
				  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

Hi @<1523701070390366208:profile|CostlyOstrich36>
Thanks a lot. It seems there is no such option in clearml-data CLI, right?

  				
Posted 
	8 months ago

					More
				  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

Write your answer

612 Views

3 Answers

8 months ago