Hello, Can Clearml.Dataset Be Put In Multiple Threads Or Processes? Especially These Two: Clearml_Dataset.Add_Files(Dst_Project_Path.Absolute()) Clearml_Dataset.Upload() Our Dataset Is About 2 Million Files, And It Is Way Too Slow

Unanswered

Hi @<1590514584836378624:profile|AmiableSeaturtle81> ! add_files already uses multi-threading, so threads would not help (see the max_workers argument).
If you are using a cloud provider such as s3 it would be useful setting this argument, or look for config entries in clearml.conf that would speed-up the upload (such as aws.s3.boto3.max_multipart_concurrency )

  				
Posted 
	6 months ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

69 Views

0 Answers

6 months ago