Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Have Been Facing This Issue "Clearml Monitor: Could Not Detect Iteration Reporting, Falling Back To Iterations As Seconds-From-Start" Due To Which The Entire Training Gets Stuck Any Ways To Solve It?

I have been facing this issue "ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start" due to which the entire training gets stuck any ways to solve it?

  
  
Posted one year ago
Votes Newest

Answers 6


if i run it without the Task.init then clearml wont run right? it works fine if i comment out this section for my main code
task = Task.init(
project_name=args.trainer.project,
task_name=args.trainer.description,
reuse_last_task_id=False,
)

task.connect_configuration(vars(args), name="args_config")

  
  
Posted one year ago

Hi @<1549202366266347520:profile|GorgeousMonkey78> , at what point does it get stuck? What happens if you remove the Task.init line from the script?

  
  
Posted one year ago

this is the basic snippet

  
  
Posted one year ago

def main(args):

# print( OmegaConf.to _yaml(args))
if args.use_clearml:
task = Task.init(
project_name=args.trainer.project,
task_name=args.trainer.description,
reuse_last_task_id=False,
)

task.connect_configuration(vars(args), name="args_config")

if __name__ == "__main__":
_config_path = "configs/yaml_files/akshay/"
_config_name = "student"

main_wrapper = hydra.main(
config_path=_config_path, config_name=_config_name, version_base="1.2"
)
main_wrapper(main)()

  
  
Posted one year ago

it gets stuck at the dataloader part before the epochs starts running

  
  
Posted one year ago

What version of clearml are you using? Can you provide a code snippet that reproduces this?

  
  
Posted one year ago