Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All — Has Anyone Run Into Degrading Download Performance With Clearml Storage Manager?

Hi all — has anyone run into degrading download performance with ClearML Storage Manager?
I’m spinning up an EC2 instance (Dockerized) with the AutoScaler and downloading ~3 TB (≈3000 files, ~1 GB each) sequentially.

  • The first files start at ~400 MB/s
  • Throughput then drops to ~100 MB/s
  • Eventually degrades to ~10 MB/sIt is really one file after another - no parallelization. What’s puzzling is that it doesn’t seem to be an instance or disk limitation: if I SSH into the same instance (while ClearML is stuck at ~100 MB/s) and use the AWS CLI to download to the same drive, I get ~300 MB/s again.
    Has anyone seen this before or found a workaround?
    image
  
  
Posted 8 days ago
pic
More
GrumpyDog7
0 × 1
Report
Votes Newest

Answers 3

Hi @<1806135344731525120:profile|GrumpyDog7> , what about the load on the server itself? Also, are you downloading from the files server or from some arbitrary source?

  
  
Posted 7 days ago
pic
More
CostlyOstrich36
0
Report
  • The server is not doing anything expect downloading
  • This is S3 data transfer within the same zone
  
  
Posted 7 days ago
pic
More
GrumpyDog7
0 × 1
Report

Hi @<1806135344731525120:profile|GrumpyDog7> , this might be related to the boto3 settings, since ClearML uses boto3 under the hood to downloads from S3. You can find the default settings here , where you can also change them using the clearml.conf configuration file (or any of the AWS boto3 env vars which boto3 can automatically detect). Also supported in this section are connect_timeout (default 60), read_timeout (default 60), stream_connections (default 128) and signature_version

  
  
Posted 6 days ago
pic
More
SuccessfulKoala55
0 × 1
Report
Write your answer
50 Views
3 Answers
8 days ago
6 days ago
Tags
clearml