MelancholyChicken65 found it ! thank you for finding this issue.
I'm hoping to get an update soon 🙂
I see, let me check the code and get back to you, this seems indeed like an issue with the Triton configuration in the model monitoring scenario.
Not sure what you mean.
If I understand the flow correclty, model auto-update
creates a monitor in model_monitoring
and then after model that complies with the monitor is found it is added to
model_monitoring_eps with specific model version and it's config. Except that it's failing because
input_name/output_name disappeared somewhere between
model_monitoring and
model_monitoring_eps ` .
Hmm is "model_monitoring_eps" another version of the model and it does not have all the properties of the "original" one?
Seems like I am using 1.2 according to https://github.com/allegroai/clearml-helm-charts/tree/main/charts/clearml-serving
And Images used in k8s cluster are:allegroai/clearml-agent-k8s-base:1.24-18 allegroai/clearml-serving-inference:1.2.0 allegroai/clearml-serving-statistics:1.2.0 allegroai/clearml-serving-triton:1.2.0-22.07 allegroai/clearml:1.7.0
MelancholyChicken65 what's the clearml-serving you are using ? (I believe this issue was fixed in 1.2)
Sure, there is a Github https://github.com/allegroai/clearml/issues/821 now :)
That looks like a bug, would you mind copy pasting this into a github issue? 🙂 AgitatedDove14 is there something else this could be?
model_monitoring
has name
but model_monitoring_eps
doesn't