Hi @<1644147961996775424:profile|HurtStarfish47> , in the ClearML SDK, all downloaded files are cached in the cache folder, the same goes for downloaded dataset files. By default, this cache is mapped to ~/.clearml/cache and contains the last 100 files downloaded.
Answered
I Am Having Trouble Understanding How Dataset Caching Works With
I am having trouble understanding how dataset caching works with clearml-data . I am using a clearml-task that runs python code to get a dataset from clearml-data . The worker is a Azure VM. I often run into issue where a dataset was downloaded on the machine, training ran for a few hours, but then dataset is no longer on the machine for validation (and the script crash and model artifact is never uploaded). How / when is dataset cache cleared ?
250 Views
1
Answer
2 months ago
2 months ago
Tags