we are at Server: 3.18.2-1126 and pypi version 1.12.2.
What's your clearml version (python and server) ?
It seems that once the job as completed once, it doesn't accept any new report...
completed can be forced, published cannot ...
What's the error you are getting ?
StrangePelican34 are you saying that after the " with " block the task is marked completed? how is that possible? is this done manually ?
It seems that once the job as completed once, it doesn't accept any new report...
AgitatedDove14 - after the model_trainer.train step it is marked as complete. This is done using our own repo - None . The extra reporting steps are not added here (I am working on that locally) but it is calling the job complete.
Hi StrangePelican34
You can either report on the Model itself:
None
or you can force it on the Task:
task = Task.get_task("task id here")
task.mark_started(force=True)
task.get_logger().report_scalar(...)
task.mark_completed(force=True)
AgitatedDove14 - for some reason none of those solutions are working. I am forcing "mark_started" - but it doesn't register. Models don't have the report_* endpoints and even trying with the artifact - once the job finishes, the artifact will no longer update.
Could it be that this is the callback that causes it?
None