Thanks for your reply and the hint to use MinIO, John!
However, you're advicing me to use --output-uri
? ClearML explicitly states the following deprecation warning when I tried that earlier:
> clearml-data create --project first_project_data --name project1 --output-uri S:\Unfinished_Learning_Data\clearml-storage
clearml-data - Dataset Management & Versioning CLI
Creating a new dataset:
Warning: '--output-uri' is deprecated, use '--storage' instead
...
Also note that the clearml-data create command accepts the --storage
parameter even though it's not listed as a parameter in the documentation you linked.
I'm getting the feeling the documentation (and possibly code?) isn't entirely on par for the deprecation of --output_uri
in favor of the --storage
parameter. What's your thoughts?
Hi @<1864479785686667264:profile|GrittyAnt2> , for that you would need to specify --output-uri
in the create command - None
This will point all previews also to the storage of your choice. Note, however as a NAS is considered part of your local disks, the browser cannot access local disk and therefor previews will not work.
For local storage solutions I suggest using something like MinIO
To iterate and fill you in with some more info:
- The data is successfully uploaded to S:\Unfinished_Learning_Data\clearml-storage - I can see it in the file system.
- I am able to download the data with a
clearml-data get --id xxxxxx
command. - When changing name of the NAS folder to S:\Unfinished_Learning_Data\clearml-storage-tempchanged, it is not possible to download data anymore - I get
Error: Could not load Dataset id=xxxxxx state
. This is good and expected. - However, with the NAS folder name changed to S:\Unfinished_Learning_Data\clearml-storage-tempchanged, it is still possible to see a preview of the PNGs in the dataset in the ClearML app interface! Cache is disabled. This is very confusing to me. It makes me worried that I don't have control of my data.