Hi Folks, Tldr: Dataset.Remove_Files() Is Very Slow. How Can I Speed It Up? I'M Working With A Large Raw Dataset That We Are Trying To Use A Small Subset Of. The Data Is Thousands Of Images And A Metadata Json File For Each Image. To Create This Subset

Unanswered

Hi Allen,
I've ran into this exact problem myself, and simply added a function to dataset.py in the clearml package ( clearml/datasets/dataset.py ) that takes a list of files instead of a single file.

It looks like this (I use clearml 1.13.1 ):

  				
Posted 
	one year ago

					More
				  		
  Report
		
					JealousMole49
				
					0
					 × 1

242 Views

0 Answers

one year ago