Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Uploaded Direct Access File To Clearml Dataset System Like This One. How Can I Access The Link Of The Uploaded Item. Whenever I Try To Call

I uploaded direct access file to clearml dataset system like this one. How can I access the link of the uploaded item. Whenever I try to call get_local_file , the function copy the origional file to cached folder. How can I prevent copying and receive back the link of those file using Dataset function? Do you guys have any suggestion?

My purpose: my dataset is really large and I just want to use clearml to save the link of training data. I don't want to real uploading, downloading or caching of the training data 🙇
image

  
  
Posted one year ago
Votes Newest

Answers 7


@<1570583237065969664:profile|AdorableCrocodile14> any file starting with file:// is considered a direct access path by default

  
  
Posted one year ago

Can you provide a code example of what you're doing?

  
  
Posted one year ago

I.e. how are you creating the dataset and how you are trying to retrieve it

  
  
Posted one year ago

@<1523701087100473344:profile|SuccessfulKoala55> The following is how I create the dataset and how I am trying to retrieve it. Is there any other way to retrieve, without actually download dataset (copying) and use the direct link access

My cache folder is /mnt/ssd2t/clearml
image
image

  
  
Posted one year ago

Hi @<1570583237065969664:profile|AdorableCrocodile14> ! get_local_copy will always copy/download external files to a folder. To get the external files, there is property on the dataset called link_entries which returns a list of LinkEntry objects, which contain a link attribute, and each such link should point to a extrenal file (in this case, your local paths prefixed with file:// )

  
  
Posted one year ago

@<1523701435869433856:profile|SmugDolphin23> Great, I am able to do it now. Thank you so much 🙇
image

  
  
Posted one year ago

@<1578193419065364480:profile|SillyLobster91> Here you go

  
  
Posted one year ago