Hi @<1541954607595393024:profile|BattyCrocodile47> , I'm happy to hear you're excited!

You certainly can replace 1 & 2 by ClearML and still use BentoML for serving. ClearML is very modular so you can select which parts you want to integrate into your existing solution. Hopefully in the end you yourself would want to fully adopt ClearML in all 4 aspects 🙂

Hope it answers the question!

That is great! This is all the motivation I needed to decide to do a POC at some point.

Hi there, I came across this post whilst searching in slack about using ClearML as a model registry - do you have any documentation or examples of this? I still can't find anything in the ClearML docs specifically about a model registry. The closest is the OutputModel class I think?

@<1557175205510516736:profile|ShallowSwan53> at this point, I think this question deserves it's own thread. I'm curious about it too!

Yes, I'm not getting very clear answers and/or can;t find anything in the documentation - there isn't a specific process for using ClearML as a model registry it seems - you just use the OutputModel class and choose your storage location...

@<1557175205510516736:profile|ShallowSwan53> , ClearML automatically logs all your models in it's model registry during your experiment's run. You can also use the OutputModel class for manual reporting as well.

Yes thanks, I know during experiments, but was hoping for something to distinguish model outputs and logging from experiments to when we want to actually publish the model in a registry as the final model.

You can actually publish models, this is a model state in ClearML. You can also use tags as an extra layer of filtering

Yes, I've figured that out too - it's just that the ClearML docs don't explicitly mention a "Model Registry" or process anywhere, as with other platforms, so it's a bit opaque and rather you figure it out by using the experiments and different functionality for tasks and put it together in your own process for a model registry. We almost didn't go with ClearML because of this, but then I suggested some steps, as you've mentioned above to help distinguish experiment tracking from a more formal model registry and we are using that now.

Examples from other platforms which make it very clear that this is a feature and have a suggested process:
None
None
None

I see, thank you for the input 🙂

We almost didn't go with ClearML because of this, but then I suggested some steps,

😞
Just in case @<1557175205510516736:profile|ShallowSwan53>
None
None
None
None

Hi friends, I'm just seeing these new messages. I read these links and I agree with @<1557175205510516736:profile|ShallowSwan53> . It's nice that the webapp has these pages, but what is the workflow to actually use this registry?

Also, @<1557175205510516736:profile|ShallowSwan53> , do you have a specific workflow in mind that you're hoping to get from ClearML?

At BEN, we're experimenting with

• BentoML for model serving. It's a Python REST framework a lot like FastAPI, but with some nice utilities for inference.
• MLFlow. We'd be open to replacing this with ClearML. I'd even push for it just for the nice integrations with the experiment tracking part of ClearML!
  Here's our MLFlow workflow

What if I wanted to do something like:

• run several experiments, logging a model artifact for each, landing in S3 as a backend
• compare the experiments and promote the best model, maybe with tags like production and click-through-rate-regressor and v1.0.0 . Advanced, stable pipelines may automate this step if the selection criteria are clear.
• Run a script from CI, possibly in Python, that- fetches the latest click-through-rate-regressor model with the production tag from the registry
• clones / installs the inference and preprocessing code that goes with that model--could be an entire REST API with that logic. Monitoring code with something like WhyLabs or Arize.ai would go in the REST API endpoints.
• builds a docker image containing the REST API and the model weights baked into the image
• deploys that image to wherever you run containers (we use AWS ECS)
  (1) and (2) seem doable to me, but (3.a) I'm not sure about.

Is there a straightforward way we could do a filter query against ClearML for model matching these criteria, and then download it?

3.a
Regarding the model query, sure from Python or restapi you can query based on any metadata
https://clear.ml/docs/latest/docs/references/sdk/model_model/#modelquery_modelsmodels

3.b
If you are using clearml-serving then check the docs / readme, but in a nutshell yes you can.
If the inference code is batchprocessing, which means a Task, then of course you can and lauch it, check the clearml agent for the launching part
3c. 3d Check clearml serving

Did I miss anything?