Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Created A Dataset With 20K Files, Total Of 20Gb, With Storage Pointing To S3. When I Upload (Or Close) The Dataset, During The Compression Phase, The Clear-Ml Cli Is Killed Due To Oom.

hi, I created a dataset with 20K files, total of 20GB, with storage pointing to S3.
When I upload (or close) the dataset, during the compression phase, the clear-ml cli is killed due to OOM.
Compressing file1.mp4 Compressing file2.mp4 Compressing file3.mp4 . . . KilledThe machine has 16 GB memory, I could always define swap memory but crashing seems excessive

  
  
Posted 2 years ago
Votes Newest

Answers 4


I can try, but it will then damage the download speeds. Anyhow not a reasonable behavior in my opinion

  
  
Posted 2 years ago

What if you reduce the chunk size using --chunk-size ? Does it affect it?

  
  
Posted 2 years ago

RAM=16
Task consumed 32GB memory total (had to add 16GB of swap)

  
  
Posted 2 years ago

Hi RoughTiger69 , how much ram is the process taking?

  
  
Posted 2 years ago
1K Views
4 Answers
2 years ago
one year ago
Tags