Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Have Been Facing This Issue "Clearml Monitor: Could Not Detect Iteration Reporting, Falling Back To Iterations As Seconds-From-Start" Due To Which The Entire Training Gets Stuck Any Ways To Solve It?

I have been facing this issue "ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start" due to which the entire training gets stuck any ways to solve it?

  
  
Posted one year ago
Votes Newest

Answers 6


this is the basic snippet

  
  
Posted one year ago

it gets stuck at the dataloader part before the epochs starts running

  
  
Posted one year ago

def main(args):

# print( OmegaConf.to _yaml(args))
if args.use_clearml:
task = Task.init(
project_name=args.trainer.project,
task_name=args.trainer.description,
reuse_last_task_id=False,
)

task.connect_configuration(vars(args), name="args_config")

if __name__ == "__main__":
_config_path = "configs/yaml_files/akshay/"
_config_name = "student"

main_wrapper = hydra.main(
config_path=_config_path, config_name=_config_name, version_base="1.2"
)
main_wrapper(main)()

  
  
Posted one year ago

Hi @<1549202366266347520:profile|GorgeousMonkey78> , at what point does it get stuck? What happens if you remove the Task.init line from the script?

  
  
Posted one year ago

if i run it without the Task.init then clearml wont run right? it works fine if i comment out this section for my main code
task = Task.init(
project_name=args.trainer.project,
task_name=args.trainer.description,
reuse_last_task_id=False,
)

task.connect_configuration(vars(args), name="args_config")

  
  
Posted one year ago

What version of clearml are you using? Can you provide a code snippet that reproduces this?

  
  
Posted one year ago