Good news:
 new 

best_model

 is saved, add a tag 

best

,
Already supported, (you just can't see the tag, but it is there :))

Interesting! Could you point me where the tagging happens? Also, by "see" do you mean UI? I tried doing task.models["outputs"][-1].tag but there's no property tag ( AttributeError: 'Model' object has no attribute 'tag' , seems that only OutputModel s have the tag property?)

I'll answer specifically for ignite on y...

I'd prefer to use config_dict, I think it's cleaner (as a workaround for metadata). However, since I'm using ignite, I think I have no way to actually do that, or at least i'm not aware of it.

But I think that whatever way one chooses, you will have to go through N best models right after training and find the best one (because of the issue we're discussing on ignite).

I think ideal would be one of the two:
Only store a single model_best . After training you just find the model with t...

Ah ok cool! Good to know thanks for clarifying 🙂 Wasn't clear if it was a bug or expected behaviour.

It doesnt work for me 😄

I'm using firefox 76.0.1 (64-bit) btw

Ah nice didn't notice that thread, will add

ah I see it. DId you add right now or was I just blind? Either way thanks for pointing 😄

So, using ignite I do the following:
` task.phases['valid'].add_event_handler(
Events.EPOCH_COMPLETED(every=1),
Checkpoint(to_save, TrainsSaver(output_uri=self.checkpoint_path), 'best', n_saved=1,
score_function=lambda x: task.phases['valid'].state.metrics[self.monitor]
if self.monitor_mode == 'max' else
-task.phases['valid'].state.metrics[self.monitor],
score_name=sel...

Great thanks!

Great, looking forward!

AgitatedDove14 we have switched to a 8 core 16 gb ram machine and haven't faced the issue since. We'll let you know if it happens. But I'm pretty confident it was the size of the machine that caused it (as I mentioned it was a 1 cpu 1.5gb ram machine)

Just checked, it did pass, training finished and all 200 models saved 🙂

Thanks for the quick responses and support too! 🙂

AgitatedDove14 Do we know what the User aborted: stopping task (3) means? It's different than when you actually abort a task yourself: User aborted: stopping task (1) .

I think that the problem happened because the VM we were using for the service queue workers was quite small (1 cpu, 1.5 gb ram), and the error message above might point to that.
We switched to a bigger one and will let you know if that was the problem.

AgitatedDove14 the funniest thing is that a train service called Allegro exists:
https://en.wikipedia.org/wiki/Allegro_(train)
Anytime I google - first result :D