Hi @<1523701111020589056:profile|DefiantSpider5>
So there are two answers here, I'll start with the open-source version of both
Is there a way in clear ml to interactively view subsets of images based on a lasso of embedding plots
The ClearML Datasets have no "query" capabilities of the data inside the dataset. That means you can see preview images, statistics and download the datasets, but no query capabilities. On the other hand, there is no limitation on the type and format of metadata, and no limitation on the formatting options for both raw data and metadata in your dataset.
The ClearML HyperDataset (paid version) has these query capabilities as well as differentiable storage on top of the metadata, raw/meta data separation etc. Unfortunately I do not think it has this specific forensics capabilities of "lasso-ing" you mentioned. I would contact sales for a more updated answer on what they do support these days.
is there a way in clear ml to calculate out metrics like recall / precision ...etc for multi class
When you create the dataset itself you can store it ontop of the dataset as part of the dateset creation tool, and this will be part of the dataset catalog. That said it is not calculated automatically as clearml is not aware of what is considered "match" in your formatting so it cannot calculate it...
is there a way in clear ml to plot a confusion matrix, click on a grid cell and see only the images that are relevant to that grid cell?
I do not think so 😞
From your question I would suggest using CleaML as a data catalog for results of your experiments (different version for each experiment), then use voxel to do live forensics on the results. This achieves both scale as the clearml dataset versioning will only store the changes, i.e. the metadata files, and the easy with the help of the CLI you will be able to easily (i.e. everything is cached) pull the results on-top your machine to further analyse with the a local run of voxel. wdyt ?
@<1523701205467926528:profile|AgitatedDove14> thanks!
yeah. i talked to their sales rep and he swore that they have feature parity with voxel, but i couldn't find anything that resembles what you refer to as forensics.
Yeah I think using voxel for forensics makes sense. What's your use case ?
I'm hoping i can find an end to end solution that also includes experiment management
I'm hoping i can find an end to end solution that also includes experiment management
Well of course biased here, but ClearML with the hyperdatasets is probably the most complete one.
Specifically with model performance analysis I would add voxel open-source to dissect specific results. but the combination of the abstraction and query capabilities of hyperdatasets, orchestration and experiment management are really unmatched for.
(and again of course I'm biased, but really there is no tool that can combine data+experiment management+orchestration on any compared level, feel free to correct me if you have thoughts )