MelancholyChicken65 found it ! thank you for finding this issue.
I'm hoping to get an update soon 🙂
I see, let me check the code and get back to you, this seems indeed like an issue with the Triton configuration in the model monitoring scenario.
Not sure what you mean.
If I understand the flow correclty, model auto-update creates a monitor in model_monitoring and then after model that complies with the monitor is found it is added to model_monitoring_eps with specific model version and it's config. Except that it's failing because input_name/output_name disappeared somewhere between model_monitoring and model_monitoring_eps ` .
Hmm is "model_monitoring_eps" another version of the model and it does not have all the properties of the "original" one?
Seems like I am using 1.2 according to https://github.com/allegroai/clearml-helm-charts/tree/main/charts/clearml-serving
And Images used in k8s cluster are:allegroai/clearml-agent-k8s-base:1.24-18 allegroai/clearml-serving-inference:1.2.0 allegroai/clearml-serving-statistics:1.2.0 allegroai/clearml-serving-triton:1.2.0-22.07 allegroai/clearml:1.7.0
MelancholyChicken65 what's the clearml-serving you are using ? (I believe this issue was fixed in 1.2)
Sure, there is a Github https://github.com/allegroai/clearml/issues/821 now :)
That looks like a bug, would you mind copy pasting this into a github issue? 🙂 AgitatedDove14 is there something else this could be?
model_monitoring has name but model_monitoring_eps doesn't