Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Am Starting To Use Clearml-Data, And I Have A Feature Request - Add A Progress Bar For The Upload Phase / Log Which Files Are Uploaded / Add Upload Speed Currently When Uploading Large Amounts Of Data, We Can An Obscure Message Of

I am starting to use clearml-data, and I have a feature request - add a progress bar for the upload phase / log which files are uploaded / add upload speed
Currently when uploading large amounts of data, we can an obscure message of Uploading compressed dataset changes ( <INT> files, total <FLOAT> GB) to http://*****:*****

  
  
Posted 3 years ago
Votes Newest

Answers 4


The issue is uploading reporting fro http uploads (object storage will report upload). Basically the http upload is post with urllib that does not support upload callbacks for progress report. If you have an idea here, we will gladly add it (as you mentioned it can be quite annoying to have to open network manager to verify the upload is progressing)

  
  
Posted 3 years ago

Thanks LethalCentipede31 , i think (3) is the most stable solution (as it doesn't require to add another package, and should work on any python version / OS)
This is actually what we do for downloads .
DO you know if there is a minimum required python requests version ?

  
  
Posted 3 years ago

I don't know

  
  
Posted 3 years ago

AgitatedDove14
option 1: use "aiohttp" (async http framework) - https://github.com/aio-libs/aiohttp
option 2: https://stackoverflow.com/questions/5925028/urllib2-post-progress-monitoring
option 3: https://docs.python-requests.org/en/latest/user/advanced/#streaming-uploads + use a generator to yield tiny chunks and print the progress between chunks

  
  
Posted 3 years ago
1K Views
4 Answers
3 years ago
one year ago
Tags