Ah ok cool! Good to know thanks for clarifying 🙂 Wasn't clear if it was a bug or expected behaviour.

AgitatedDove14 we have switched to a 8 core 16 gb ram machine and haven't faced the issue since. We'll let you know if it happens. But I'm pretty confident it was the size of the machine that caused it (as I mentioned it was a 1 cpu 1.5gb ram machine)

Just checked, it did pass, training finished and all 200 models saved 🙂

Great thanks!

It doesnt work for me 😄

I'm using firefox 76.0.1 (64-bit) btw

Thanks for the quick responses and support too! 🙂

Great, looking forward!

Ah nice didn't notice that thread, will add

So, using ignite I do the following:
` task.phases['valid'].add_event_handler(
Events.EPOCH_COMPLETED(every=1),
Checkpoint(to_save, TrainsSaver(output_uri=self.checkpoint_path), 'best', n_saved=1,
score_function=lambda x: task.phases['valid'].state.metrics[self.monitor]
if self.monitor_mode == 'max' else
-task.phases['valid'].state.metrics[self.monitor],
score_name=sel...

Good news:
 new 

best_model

 is saved, add a tag 

best

,
Already supported, (you just can't see the tag, but it is there :))

Interesting! Could you point me where the tagging happens? Also, by "see" do you mean UI? I tried doing task.models["outputs"][-1].tag but there's no property tag ( AttributeError: 'Model' object has no attribute 'tag' , seems that only OutputModel s have the tag property?)

I'll answer specifically for ignite on y...

ah I see it. DId you add right now or was I just blind? Either way thanks for pointing 😄

AgitatedDove14 the funniest thing is that a train service called Allegro exists:
https://en.wikipedia.org/wiki/Allegro_(train)
Anytime I google - first result :D

I'd prefer to use config_dict, I think it's cleaner (as a workaround for metadata). However, since I'm using ignite, I think I have no way to actually do that, or at least i'm not aware of it.

But I think that whatever way one chooses, you will have to go through N best models right after training and find the best one (because of the issue we're discussing on ignite).

I think ideal would be one of the two:
Only store a single model_best . After training you just find the model with t...

AgitatedDove14 Do we know what the User aborted: stopping task (3) means? It's different than when you actually abort a task yourself: User aborted: stopping task (1) .

I think that the problem happened because the VM we were using for the service queue workers was quite small (1 cpu, 1.5 gb ram), and the error message above might point to that.
We switched to a bigger one and will let you know if that was the problem.