Also I do get new data everyday for classification task, how do I tune my model everyday on that new data?

That means that in inference you are reporting the F1 score, correct?

For classification it's F1 score but for other task it maybe and I don't think that's problem. we just have to log it right?

Are you using passing output_uri to Task.init?

No

are you storing the model as artifact?

Yes

Hi AgitatedDove14
I have multiple models for a classification task but over the time performance is degrading. I want to trigger a retrain task when F1 score drops below certain value. As part of the retraining I have to train all the models and then have to choose best one and deploy it. How do I achieve this using trains?

Give me few days, I will work on your sugestions and then let you know if I am not able to do this.