@<1523701205467926528:profile|AgitatedDove14> - for some reason none of those solutions are working. I am forcing "mark_started" - but it doesn't register. Models don't have the report_* endpoints and even trying with the artifact - once the job finishes, the artifact will no longer update.
What's your clearml version (python and server) ?
It seems that once the job as completed once, it doesn't accept any new report...
completed can be forced, published cannot ...
What's the error you are getting ?
It seems that once the job as completed once, it doesn't accept any new report...
we are at Server: 3.18.2-1126 and pypi version 1.12.2.
@<1523711619815706624:profile|StrangePelican34> are you saying that after the " with " block the task is marked completed? how is that possible? is this done manually ?
Hi @<1523711619815706624:profile|StrangePelican34>
You can either report on the Model itself:
None
or you can force it on the Task:
task = Task.get_task("task id here")
task.mark_started(force=True)
task.get_logger().report_scalar(...)
task.mark_completed(force=True)
@<1523701205467926528:profile|AgitatedDove14> - after the model_trainer.train step it is marked as complete. This is done using our own repo - None . The extra reporting steps are not added here (I am working on that locally) but it is calling the job complete.
Could it be that this is the callback that causes it?
None