Great thanks!

So, using ignite I do the following:
` task.phases['valid'].add_event_handler(
Events.EPOCH_COMPLETED(every=1),
Checkpoint(to_save, TrainsSaver(output_uri=self.checkpoint_path), 'best', n_saved=1,
score_function=lambda x: task.phases['valid'].state.metrics[self.monitor]
if self.monitor_mode == 'max' else
-task.phases['valid'].state.metrics[self.monitor],
score_name=sel...

Good news:
 new 

best_model

 is saved, add a tag 

best

,
Already supported, (you just can't see the tag, but it is there :))

Interesting! Could you point me where the tagging happens? Also, by "see" do you mean UI? I tried doing task.models["outputs"][-1].tag but there's no property tag ( AttributeError: 'Model' object has no attribute 'tag' , seems that only OutputModel s have the tag property?)

I'll answer specifically for ignite on y...

AgitatedDove14 Do we know what the User aborted: stopping task (3) means? It's different than when you actually abort a task yourself: User aborted: stopping task (1) .

I think that the problem happened because the VM we were using for the service queue workers was quite small (1 cpu, 1.5 gb ram), and the error message above might point to that.
We switched to a bigger one and will let you know if that was the problem.

I'd prefer to use config_dict, I think it's cleaner (as a workaround for metadata). However, since I'm using ignite, I think I have no way to actually do that, or at least i'm not aware of it.

But I think that whatever way one chooses, you will have to go through N best models right after training and find the best one (because of the issue we're discussing on ignite).

I think ideal would be one of the two:
Only store a single model_best . After training you just find the model with t...

Great, looking forward!

Just checked, it did pass, training finished and all 200 models saved 🙂

Thanks for the quick responses and support too! 🙂

AgitatedDove14 the funniest thing is that a train service called Allegro exists:
https://en.wikipedia.org/wiki/Allegro_(train)
Anytime I google - first result :D

It doesnt work for me 😄

I'm using firefox 76.0.1 (64-bit) btw

ah I see it. DId you add right now or was I just blind? Either way thanks for pointing 😄

AgitatedDove14 we have switched to a 8 core 16 gb ram machine and haven't faced the issue since. We'll let you know if it happens. But I'm pretty confident it was the size of the machine that caused it (as I mentioned it was a 1 cpu 1.5gb ram machine)

Ah ok cool! Good to know thanks for clarifying 🙂 Wasn't clear if it was a bug or expected behaviour.

Ah nice didn't notice that thread, will add