Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
After I Finish Training A Model, I Want To Call Logger.Report_Scalars To Help Monitor Inferencing Status (We Do A Lot Of Batch) But After The Model Finishes Training, Scalars Are No Longer Accepted By The Task As It Is Considered Completed. Help!

After I finish training a model, I want to call logger.report_scalars to help monitor inferencing status (we do a lot of batch) but after the model finishes training, scalars are no longer accepted by the task as it is considered completed. Help!

  
  
Posted 7 months ago
Votes Newest

Answers 9


@<1523711619815706624:profile|StrangePelican34> are you saying that after the " with " block the task is marked completed? how is that possible? is this done manually ?

  
  
Posted 6 months ago

Hi @<1523711619815706624:profile|StrangePelican34>
You can either report on the Model itself:
None
or you can force it on the Task:

task = Task.get_task("task id here")
task.mark_started(force=True)
task.get_logger().report_scalar(...)
task.mark_completed(force=True)
  
  
Posted 7 months ago

What's your clearml version (python and server) ?

It seems that once the job as completed once, it doesn't accept any new report...

completed can be forced, published cannot ...
What's the error you are getting ?

  
  
Posted 6 months ago

we are at Server: 3.18.2-1126 and pypi version 1.12.2.

  
  
Posted 6 months ago

It seems that once the job as completed once, it doesn't accept any new report...

  
  
Posted 6 months ago

@<1523701205467926528:profile|AgitatedDove14> - for some reason none of those solutions are working. I am forcing "mark_started" - but it doesn't register. Models don't have the report_* endpoints and even trying with the artifact - once the job finishes, the artifact will no longer update.

  
  
Posted 6 months ago

@<1523701205467926528:profile|AgitatedDove14> - after the model_trainer.train step it is marked as complete. This is done using our own repo - None . The extra reporting steps are not added here (I am working on that locally) but it is calling the job complete.

  
  
Posted 6 months ago

I opened an issue here: None

  
  
Posted 6 months ago

Could it be that this is the callback that causes it?
None

  
  
Posted 6 months ago
494 Views
9 Answers
7 months ago
6 months ago
Tags