Reputation
Badges 1
6 × Eureka!The question is — are there any workarounds to set last iteration to correct value. And preferably do it in a simple way (i.e. not setting it manually).
I tried it, but unfortunately, this way it only sets last iteration to 0 instead of using last iteration from TensorBoard and simply rewrites logs. Expected behaviour is that it reads last iteration correctly. At least it is stated in docs so.
Thanks Martin. I tried to rerun everything from scratch using continue_last_task=0
and looks like it helped a lot but not completely. You can see in attached screenshot that gaps in iteration axis are still a little bigger than expected. I’v rerun it two times.
No, I don’t need last iteration set to zero. All I need is to ClearML correctly initialize it from TensorBoard (or from wherever it initializes it). When I train model, stop training and then resume it, ClearML instead of using last iteration doubles (I guess) it. And this can be seen in attached screenshot in GitHub issue.
Hi Martin, thanks for the response! Nope, setting initial iteration didn’t solve the problem.
Thanks a lot for your help!