Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello! I Have A Dataset On A /Mnt Share. When I Try To Get A Local Copy, The Dataset On The Share Is Deleted. Is This Correct Behaviour? This Is How I Get The Dataset:

Hello! I have a dataset on a /mnt share. When I try to get a local copy, the dataset on the share is deleted. Is this correct behaviour? This is how I get the dataset:
data = clearml.Dataset.get(dataset_id='7873a3b0764145f086a2af139c5f1fc9') path = data.get_local_copy(use_soft_links=False)After calling .get_local_copy() all the zipped files in artifact data folders are deleted. I would like for the dataset to remain on the share, as I use it accross multiple machines to just train and not modify. This is how I create the dataset:
` dataset = clearml.Dataset.create(dataset_name='cool-dataset', dataset_project='my-project',
dataset_version="1.0.0")

dataset.add_files(str(data_path), recursive=True)
dataset.upload(output_url='/mnt/machine_learning/datasets')
dataset.finalize() `

  
  
Posted 2 years ago
Votes Newest

Answers 16


1.7.1

  
  
Posted 2 years ago

Hi DeliciousKoala34 ,

dataset on the share is deleted

the dataset task or the data in the folder? both shouldnt be deleted by get_local_copy activation

  
  
Posted 2 years ago

which clearml version do you use?

  
  
Posted 2 years ago

TimelyPenguin76
data in the folder, the dataset task is not deleted

  
  
Posted 2 years ago

Unfortunately, same thing happens

  
  
Posted 2 years ago

sorry, should have specified better
full path: /mnt/machine_learning/datasets/my-project/.datasets/cool-dataset/cool-dataset.7873a3b0764145f086a2af139c5f1fc9/artifacts

  
  
Posted 2 years ago

DeliciousKoala34 can you upgrade to clearml==1.8.0 ? the issue should be fixed now

  
  
Posted 2 years ago

The state folder is not affected

this is /mnt/machine_learning/datasets folder?

  
  
Posted 2 years ago

Hi DeliciousKoala34 . I was able to reproduce your issue. I'm now looking for a solution for your problem. Thank you

  
  
Posted 2 years ago

So which data is being deleted? which folder is the “artifact folder”?

  
  
Posted 2 years ago

SmugDolphin23 Yes it works now! Thank you both very much for fast and great support!

  
  
Posted 2 years ago

SmugDolphin23 Thanks!

  
  
Posted 2 years ago

The dataset is downloaded to my local clearml cache. But specifically all of the contents of the data folders, under artifacts folder are removed. The state folder is not affected.

  
  
Posted 2 years ago

can you try with the latest?
pip install clearml==1.7.3rc1

  
  
Posted 2 years ago

SmugDolphin23 Hello, sorry to bother, but are there any updates on this issue?

  
  
Posted 2 years ago

And in that folder there are 5 data folders, I assume for every compressed chunk, and a state folder

  
  
Posted 2 years ago