Hi, It seems like when I upload the dataset to ClearML with the methods from the Dataset class. It packed the whole dataset into several .zip files in /tmp folder before uploading. I'd like to know whether I can point the .zip files to be saved somewhere else rather than /tmp folder cuz, for the large dataset, i cannot store these .zip files in the local machine. So, i want to point them out to be saved in some mounted directory instead.
If you set the TMP env variable you can control the tmp folder. Would that work?
Thanks AgitatedDove14 , I tried setting TEMP environment variable to another directory and it worked! Not sure whether TMP could work too.
Hi DeterminedOwl36 , what about simply adding files that are already sitting on some storage?
https://clear.ml/docs/latest/docs/references/sdk/dataset#add_external_files
