Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Here'S My Question. I Have A Free Account On App.Clear.Ml And I'M Trying To Create A Dataset Consisting Of Images And Masks (Jpgs And Pngs). I Use The Following Commands:

Here's my question. I have a free account on app.clear.ml and I'm trying to create a dataset consisting of images and masks (jpgs and pngs). I use the following commands:

clearml-data create --project first_project_data --name project1 --storage S:\Unfinished_Learning_Data\clearml-storage

followed by

clearml-data sync --folder C:\data\learning_data\project1 --storage S:\Unfinished_Learning_Data\clearml-storage

Where the S:\ disk is a NAS (network storage), accessible from several servers in my network.

Indeed when upload is completed, I can see the data uploaded to S:\Unfinished_Learning_Data\clearml-storage\first_project_data.datasets\project1

So far so good!

However, when I try to make the data inaccessible to the ClearML app interface by changing name of the folder on the S:\ disk, I can still Preview the images in the Dataset view! I can also Download the images from the interface. Looking in DevTools, I can see in the Request URL that it points to a location on None ...

This is all very confusing to me. If I upload files specifying a local --storage location, I would not expect the files to end up on ClearML servers. In fact I really don't want this - I want to keep my data local.

Can someone help me understand if I've misunderstood something or what I can do to reach my goal here?

  
  
Posted 24 days ago
Votes Newest

Answers 3


Thanks for your reply and the hint to use MinIO, John!

However, you're advicing me to use --output-uri ? ClearML explicitly states the following deprecation warning when I tried that earlier:

> clearml-data create --project first_project_data --name project1  --output-uri S:\Unfinished_Learning_Data\clearml-storage
clearml-data - Dataset Management & Versioning CLI
Creating a new dataset:
Warning: '--output-uri' is deprecated, use '--storage' instead
...

Also note that the clearml-data create command accepts the --storage parameter even though it's not listed as a parameter in the documentation you linked.

I'm getting the feeling the documentation (and possibly code?) isn't entirely on par for the deprecation of --output_uri in favor of the --storage parameter. What's your thoughts?

  
  
Posted 24 days ago

Hi @<1864479785686667264:profile|GrittyAnt2> , for that you would need to specify --output-uri in the create command - None

This will point all previews also to the storage of your choice. Note, however as a NAS is considered part of your local disks, the browser cannot access local disk and therefor previews will not work.

For local storage solutions I suggest using something like MinIO

  
  
Posted 24 days ago

To iterate and fill you in with some more info:

  • The data is successfully uploaded to S:\Unfinished_Learning_Data\clearml-storage - I can see it in the file system.
  • I am able to download the data with a clearml-data get --id xxxxxx command.
  • When changing name of the NAS folder to S:\Unfinished_Learning_Data\clearml-storage-tempchanged, it is not possible to download data anymore - I get Error: Could not load Dataset id=xxxxxx state . This is good and expected.
  • However, with the NAS folder name changed to S:\Unfinished_Learning_Data\clearml-storage-tempchanged, it is still possible to see a preview of the PNGs in the dataset in the ClearML app interface! Cache is disabled. This is very confusing to me. It makes me worried that I don't have control of my data.
  
  
Posted 24 days ago
138 Views
3 Answers
24 days ago
23 days ago
Tags
Similar posts