I'd prefer to use config_dict, I think it's cleaner (as a workaround for metadata). However, since I'm using ignite, I think I have no way to actually do that, or at least i'm not aware of it.

But I think that whatever way one chooses, you will have to go through N best models right after training and find the best one (because of the issue we're discussing on ignite).

I think ideal would be one of the two:
Only store a single model_best . After training you just find the model with that name in task.models["output"] Store as many model_best as you want (with n_saved from ignite). Everytime a new best_model is saved, add a tag best , and remove the tag from the previous best models (not sure how straightforward that is though)So one can just do at the end best = task.get_model_best After that you could also programmatically check against the accuracy of your production model (using metadata / json config) and decide what youwant to do with the model .

what do you think?