DilapidatedDucks58 by default if you continue to execution, it will automatically continue reporting from the last iteration . I think this is what you are seeing

😞 DilapidatedDucks58 how exactly are you "relaunching/continue" the execution? And what exactly are you setting?

maybe I should use explicit reporting instead of Tensorboard

It will do just the same 😞

there is no method for setting 

last iteration

, which is used for reporting when continuing the same task. maybe I could somehow change this value for the task?

Let me double check that...

overwriting this value is not ideal though, because for :monitor:gpu and :monitor:machine ...

That is a very good point

but for the metrics, I explicitly pass the number of epoch that my training is currently on...

Yes so the idea it already "knows" where you stopped, so when you are reporting "iteration 1" it knows it's actually 0+previous_last_iteration ...

I think we should open a GitHub Issue and get some more feedback, maybe we should just add support in the backend side ?

Thank you DilapidatedDucks58 for the ping!
totally slipped my mind 😞

okay, so if there’s no workaround atm, should I create a Github issue?

this would be great. I could just then pass it as a hyperparameter

nope, didn't work =(

Many thanks!

overwriting this value is not ideal though, because for :monitor:gpu and :monitor:machine values I would like to continue from the latest iteration

but for the metrics, I explicitly pass the number of epoch that my training is currently on. it'ls kind of weird that it adds offset to the values that are explicitly reported, no?

task = Task.get_task(task_id = args.task_id)
task.mark_started()
task.set_parameters_as_dict(
{
"General": {
"checkpoint_file": model.url,
"restart_optimizer": False,
}
}
)
task.set_initial_iteration(0)
task.mark_stopped()
Task.enqueue(task = task, queue_name = task.data.execution.queue)

Hi DilapidatedDucks58
apologies, this thread slipped way.
I double checked, there server will not allow you to overwrite it (meaning to have it fixed will need to release a server version which usually takes longer)
That said maybe we can pass an argument to the "Task.init" so it ignores it? wdyt?

Hmm I suspect the 'set_initial_iteration' does not change/store the state on the Task, so when it is launched, the value is not overwritten. Could you maybe open a GitHub issue on it?

there is no method for setting last iteration , which is used for reporting when continuing the same task. maybe I could somehow change this value for the task?

it will probably screw up my resource monitoring plots, but well, who cares 😃

still no luck, I tried everything =( any updates?

not sure what you mean. I used to do task.set_initial_iteration(task.get_last_iteration()) in the task resuming script, but in the training code I explicitly pass global_step=epoch to the TensorBoard writer

sorry that I keep bothering you, I love ClearML and try to promote it whenever I can, but this thing is a real pain in the ass 😃

this is how it looks if I zoom in on the epochs that ran before the crash

thank you, I'll let you know if setting it to zero worked

maybe I should use explicit reporting instead of Tensorboard

perhaps I need to do task.set_initial_iteration(0)?

https://github.com/allegroai/clearml/issues/496

okay, I will open an issue

sorry that I keep bothering you, I love ClearML and try to promote it whenever I can, but this thing is a real pain in the ass 

No worries I totally feel you.
As a quick hack in the actual code of the Task itself, is it reasonable to have:
task = Task.init(....) task.set_initial_iteration(0)

Lol, :)
I think the issue is that you do not need to manually set the initial iteration, it's supposed to get it , as it is stored on the Task itself

Yep it should :)
I assume you add the previous iteration somewhere else, and this is the cause for the issue?

I use Docker for training, which means that log_dir contents are removed for the continued experiment btw

does this mean that setting initial iteration to 0 should help?