I am currently running a Clearml-server and have few questions about Dataset management.
https://clear.ml/docs/latest/docs/getting_started/ds/best_practices#train-remotely This is the situation I am currently in where i would like to develop locally and train remotely. What i have done so far is that, i have a custom_dataset_large(lets assume that the original dataset is 200Gb) and a mini version of it(less than 5Gb). I have the custom_dataset_large uploaded into the file_server via clearml-data and i have a dataset id for it. The dataset comprises of a bunch of directories which inturn contain data upon which a model needs to be trained. How can i parse the custom_dataset_large without downloading a local cache of it. cause the only way i see it is via Dataset.get_local_copy() method, which i dont want to do as it download the complete dataset into the local dev machine which doesnt have that much storage. What i am looking for is, how to fetch the remote dataset url path which I uploaded via cleaml-data to.