By the way, will downloading still happen if the datasets is available in the cache folder? Any specific settings to add to Dataset.get_local_copy()?
By the way, will downloading still happen if the datasets is available in the cache folder?
If it is cached, then there is no need to re-download 🙂
When you set the pod make sure you mount the clearml local cache folder to the PV
basically /root/.clearml/cache/
I have yet to figure out how to do so, would appreciate if u could give some guidance
Hi @<1523701304709353472:profile|OddShrimp85>
Do you mean Dataset.get_local_copy()
?
@<1523701205467926528:profile|AgitatedDove14> when my codes get the clearml datasets, it stores in the cache e.g. /$HOME/.clearml/cache....
I wanted it to be in a mounted PV instead, so other pods (in same node) who needed same datasets can use without pulling again.