We have some scenario where a group of clearml experiments might represent a logical experiment. We then want to use all the trained models in a pipeline to generate some output.

With that output, we probably want to some third party like mechanical turk, do some custom evaluations - and some times more than once. We then want to connect (and present) these evaluations alongwith ClearML experiments.

we have various services internally to do this --> however, we have to manually link it up with some clearml experiments. Just wondering if there is an extension or plugin API that clearml provides so we can present this to the data scientist in a unified way?

Hi @<1535069219354316800:profile|PerplexedRaccoon19> , there's no such plugin at the moment, but to automate it would be pretty easy using the SDK or REST API calls, I think

Hi @<1535069219354316800:profile|PerplexedRaccoon19> , not sure I understand what you mean, can you please elaborate on what you mean by doing the evaluations within ClearML?