By the way, will downloading still happen if the datasets is available in the cache folder? Any specific settings to add to Dataset.get_local_copy()?

By the way, will downloading still happen if the datasets is available in the cache folder?

If it is cached, then there is no need to re-download 🙂

When you set the pod make sure you mount the clearml local cache folder to the PV
basically /root/.clearml/cache/

I have yet to figure out how to do so, would appreciate if u could give some guidance

OK let me try by adding to vol mount.

Hi @<1523701304709353472:profile|OddShrimp85>
Do you mean Dataset.get_local_copy() ?

@<1523701205467926528:profile|AgitatedDove14> when my codes get the clearml datasets, it stores in the cache e.g. /$HOME/.clearml/cache....

I wanted it to be in a mounted PV instead, so other pods (in same node) who needed same datasets can use without pulling again.