@<1523701205467926528:profile|AgitatedDove14> when my codes get the clearml datasets, it stores in the cache e.g. /$HOME/.clearml/cache....
I wanted it to be in a mounted PV instead, so other pods (in same node) who needed same datasets can use without pulling again.
By the way, will downloading still happen if the datasets is available in the cache folder?
If it is cached, then there is no need to re-download 🙂
When you set the pod make sure you mount the clearml local cache folder to the PV
basically /root/.clearml/cache/
I have yet to figure out how to do so, would appreciate if u could give some guidance
By the way, will downloading still happen if the datasets is available in the cache folder? Any specific settings to add to Dataset.get_local_copy()?
Hi @<1523701304709353472:profile|OddShrimp85>
Do you mean Dataset.get_local_copy() ?