Unanswered
Can You Help Me Make The Case For Clearml Pipelines/Tasks Vs Metaflow?
Context Within...
For these functions, Metaflow offers:
- triggering: integration with AWS event bridge. It's really easy to use Boto3 and AWS access keys to emit events for Metaflow DAGs. It's nice not to have to worry about networking for this.
- Scheduling: The fact that Metaflow uses stepfunctions is reassuring.
- observability: this lovely flame graph where you can view the logs and duration of each step in the DAG, it's easy to view all the DAG runs including the ones that have failed. Ideally, we would be able to see the status of all of our pipelines in a single UI.
- alerts: it's easy to set up alerts for all DAGs at once. Actually, this may not be set up the way I imagine. But I really want- Data scientists to author their own pipelines
- Data scientists not to have to worry / understand how to set up alerts for failed tasks/pipelines
- Every pipeline to be set up with alerts--maybe this is just as hard with Metaflow as it is with ClearML.
Is there a low-effort way to set all these things up with ClearML open source or enterprise?
158 Views
0
Answers
one year ago
one year ago