Hi team,
I'm having an issue with ClearML Serving custom metrics not showing up in Prometheus or Grafana. Here's what I did:

• Added custom metrics with the following command:clearml-serving --id d620517bd02d4aad8fed21309c54da3e metrics add --endpoint "text_classifier6" \
--variable-scalar "request_count=0,100,1000,10000" \
--variable-scalar "batch_size=1,10,50,100" \
--variable-scalar "processing_time_seconds=0.01,0.1,0.5,1,5" \
--variable-scalar "error_rate=0,0.01,0.05,0.1,0.5" \
--variable-value "total_items_processed" \
--variable-value "average_processing_time" \
--variable-value "error_count" \
--log-freq 1.0

2. Implemented collect_custom_statistics_fn in the preprocessing script with the following:

collect_custom_statistics_fn({
"processing_time_seconds": processing_time,
"batch_size": len(data),
"average_processing_time": self.total_processing_time / self.total_items_processed,
"total_items_processed": self.total_items_processed,
"successful_predictions": self.total_items_processed,
"error_rate": (self.error_count / self.request_count) if self.request_count > 0 else 0
})

I can only see default metrics in Prometheus ( _count_created , _latency_bucketcounter , etc.), but none of the custom metrics like average_processing_time or total_items_processed .
Has anyone encountered this or knows how to fix it? Thanks!