MelancholyChicken65 found it ! thank you for finding this issue.
I'm hoping to get an update soon 🙂
MelancholyChicken65 what's the clearml-serving you are using ? (I believe this issue was fixed in 1.2)
Not sure what you mean.
If I understand the flow correclty, model auto-update
creates a monitor in model_monitoring
and then after model that complies with the monitor is found it is added to
model_monitoring_eps with specific model version and it's config. Except that it's failing because
input_name/output_name disappeared somewhere between
model_monitoring and
model_monitoring_eps ` .
Hmm is "model_monitoring_eps" another version of the model and it does not have all the properties of the "original" one?
Seems like I am using 1.2 according to https://github.com/allegroai/clearml-helm-charts/tree/main/charts/clearml-serving
And Images used in k8s cluster are:allegroai/clearml-agent-k8s-base:1.24-18 allegroai/clearml-serving-inference:1.2.0 allegroai/clearml-serving-statistics:1.2.0 allegroai/clearml-serving-triton:1.2.0-22.07 allegroai/clearml:1.7.0
That looks like a bug, would you mind copy pasting this into a github issue? 🙂 AgitatedDove14 is there something else this could be?
Sure, there is a Github https://github.com/allegroai/clearml/issues/821 now :)
model_monitoring
has name
but model_monitoring_eps
doesn't
I see, let me check the code and get back to you, this seems indeed like an issue with the Triton configuration in the model monitoring scenario.