Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All

Hi all 🙂 ,
There is a way to freeze the iteration monitoring?

I like to download my data/model at the start of my code, but it takes a while and I am getting
ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start
There is a way to tell ClearML to wait while downloading?

  
  
Posted 3 years ago
Votes Newest

Answers 6


Thanks for the quick replay.
This will set more time before the timeout right?

Maybe there is a way to do something like:
task.freeze_monitor() download() task.defrost_monitor()

  
  
Posted 3 years ago

The main reason to add the timeout is because the warning was annoying to users 🙂
The secondary was that clearml will start reporting based on seconds from start, then when iterations start it will revert back to iterations. But if the iterations are "epochs" the numbers are lower so you end up with a graph that does not match the expected "iterations" x-axis. Make sense ?

  
  
Posted 3 years ago

This will set more time before the timeout right?

Correct.

task.freeze_monitor()
download()
task.defrost_monitor()

Currently there isn't, but that's a good ides.
What would be the argument of using it vs increasing the timeout ?
btw: setting the resource timeout to 99999 will basically mean that it will wait until the first reported iteration, Not that it will just sleep for 99999sec 🙂

  
  
Posted 3 years ago

OK thanks for the answer.. I will use
task.set_resource_monitor_iteration_timeout(seconds_from_start=1800)as you suggested for now..

If you will add something like I suggest can you notify me?

  
  
Posted 3 years ago

I am sure you add this timeout for a reason.

Probably since increasing the timeout can affect other functionality. .

Am I wrong?

  
  
Posted 3 years ago

Hi CooperativeFox72
Sure 🙂
task.set_resource_monitor_iteration_timeout(seconds_from_start=1800)

  
  
Posted 3 years ago