Unanswered
Hi Folks,
Tldr: Dataset.Remove_Files() Is Very Slow. How Can I Speed It Up?
I'M Working With A Large Raw Dataset That We Are Trying To Use A Small Subset Of. The Data Is Thousands Of Images And A Metadata Json File For Each Image. To Create This Subset
Hi Allen,
I've ran into this exact problem myself, and simply added a function to dataset.py
in the clearml package ( clearml/datasets/dataset.py
) that takes a list of files instead of a single file.
It looks like this (I use clearml 1.13.1
):
86 Views
0
Answers
7 months ago
7 months ago