CostlyOstrich36 Hi,
I have a question related to ClearML’s indexing mechanism for cached datasets. We noticed that when storing the dataset cache folder on an NFS (Network File System), running the command clearml-data get triggers a cache indexing process, which takes a significant amount of time. However, if we remove the NFS cache folder, the command runs almost instantly.
Could you explain how caching works in ClearML? Specifically:

• Why does ClearML perform global folder indexing before the script starts?
• Why does it index the dataset cache folder when executing clearml-data get ?
• Is there an option to disable cache indexing or control its behavior to optimize performance, especially when using NFS?Any insights or workarounds to speed up the process would be greatly appreciated.
  Thanks!