By the way, will downloading still happen if the datasets is available in the cache folder?
If it is cached, then there is no need to re-download 🙂
I have yet to figure out how to do so, would appreciate if u could give some guidance
Hi @<1523701304709353472:profile|OddShrimp85>
Do you mean Dataset.get_local_copy() ?
@<1523701205467926528:profile|AgitatedDove14> when my codes get the clearml datasets, it stores in the cache e.g. /$HOME/.clearml/cache....
I wanted it to be in a mounted PV instead, so other pods (in same node) who needed same datasets can use without pulling again.
When you set the pod make sure you mount the clearml local cache folder to the PV
basically /root/.clearml/cache/
By the way, will downloading still happen if the datasets is available in the cache folder? Any specific settings to add to Dataset.get_local_copy()?