TenseOstrich47 , you could create a monitor task that reads model performance from your database and reports them as some scalar. According to that scalar you can create triggers 🙂
What do you think?
What do you mean? Do you have a reference?
but have our own custom logging of metrics etc.
Are those custom metrics reported to the ClearML server or stored somewhere else?
Just wondering how tricky integrating a trigger would be for that
I guess it really depends on your current implementation currently
Hi TenseOstrich47 What you can do is report the metric to clearml, then use the Taskscheduler to listen on a specific project. If a task in this project reports a metric below \ above a certain TH (Or I think if it's the highest \ lowest as well) you can trigger an event (Task \ function). That's how you do it with the Taskscheduler object
Hi TenseOstrich47 ,
Our model store consists of metadata stored in the DWH, and model artifacts stored in S3. We technically use ClearML for managing the hardware resource for running experiments, but have our own custom logging of metrics etc. Just wondering how tricky integrating a trigger would be for that
Say we have a DAG running on airflow every 30 mins. The purpose of this DAG is to aggregate results of model performance. If model performance is poor, then it sends a message to a queue with some config on which model to re-train.
I would like to use a TaskScheduler to poll this queue every X interval, to check whether a training pipeline needs to be kickstarted or not
Hi TenseOstrich47 Yup 🙂 You can check our scheduler module:
It supports time-events as well as triggers to external events