Hi, It seems like when I upload the dataset to ClearML with the methods from the Dataset class. It packed the whole dataset into several .zip files in /tmp folder before uploading. I'd like to know whether I can point the .zip files to be saved somewhere else rather than /tmp folder cuz, for the large dataset, i cannot store these .zip files in the local machine. So, i want to point them out to be saved in some mounted directory instead.
Hi DeterminedOwl36 , what about simply adding files that are already sitting on some storage?
https://clear.ml/docs/latest/docs/references/sdk/dataset#add_external_files
If you set the TMP env variable you can control the tmp folder. Would that work?
Thanks AgitatedDove14 , I tried setting TEMP environment variable to another directory and it worked! Not sure whether TMP could work too.
