Ok, good to know! Thank you very much for doing this!

Hey @<1541592213111181312:profile|PleasantCoral12> thanks for doing the profiling! This looks pretty normal to me. Although 37 seconds for a dataset.get is definitely too much. I just checked and for me it takes 3.7 seconds. Mind you the .get() method doesn't actually download the data, so the dataset size is irrelevant here.

But the slowdowns do seem to only occur when doing api requests. Possible next steps could be:

• Send me your username and email address (maybe dm if you don't wa...

AdventurousButterfly15 The fact that it tries to ping localhost means you are running the ClearML server locally right? In that case, it is a docker thing: it cannot access localhost because localhost inside a docker image is not the same one as your machine itself. They're isolated.

That said, adding --network=host to the docker command usually fixes this by connecting the container to the local network instead of the internal docker one.

You can add a custom argument either i...

effectively making us lose 24 hours of GPU compute

Oof, sorry about that, man 😞

Just for reference, the main issue is that ClearML does not allow non-string types as dict keys for its configuration. Usually the labeling mapping does have ints as keys. Which is why we need to cast them to strings first, then pass them to ClearML then cast them back.

Hey @<1539780272512307200:profile|GaudyPig83> !

Are you running a self-hosted server? Is this the only type of HTTP call that fails or does e.g. logging experiments also not work? A connection error usually means your docker containers can't reach each other.

Hi CurvedHedgehog15 , so my previous reply does assume you have reported a scalar for each individual FAR level. Then you can add individual levels as shown in the gif. But like you siad, that might actually cause you to loose your overview in the scalars tab.

So I don't think there's an immediate way to do this in ClearML right now, but would you mind opening an issue on github for it? It might be interesting to add it to the tool?

No inputs and outputs are ever set automatically 🙂 For e.g. Keras you'll have to specify it using the CLI when making the endpoint, so Triton knows how to optimise as well as set it correctly in your preprocessing so Triton receives the format it expects.

Sure! This is an example of running a custom model. It basically boils down to defining a preprocess, process and postprocess function. Inside the process function can be anything, including just a basic call to huggingface to run inference 🙂
I have not tested this myself mind you, but I see no reason why it wouldn't work!
In fact, I think even Triton itself supports running on CPU these days, so you still ...

Can you elaborate a bit more, I don't quite understand yet. So it works when you update an existing task by adding a tag to it, but it doesn't work when adding a tag for the first time?

Hi @<1547028116780617728:profile|TimelyRabbit96> Awesome that you managed to get it working!

To be honest, I'm not completely sure as I've never tried hundreds of endpoints myself. In theory, yes it should be possible, Triton, FastAPI and Intel OneAPI (ClearML building blocks) all claim they can handle that kind of load, but again, I've not tested it myself.

To answer the second question, yes! You can basically use the "type" of model to decide where it should be run. You always have the custom model option if you want to run it yourself too 🙂

No worries! And thanks for putting in the time.

Could you try and create a new task with the tag already added? Instead of adding a tag on an existing task. It should work then. If it does, this might be a bug? Or if not, a good feature to exist 🙂

An update: using your code (the snippet above) I was getting no scalars when simply installing ultralytics and clearml packages using pip. Because indeed tensorboard is not installed. When I do install tensorboard, I get metrics in like normal, so I can't seem to reproduce the issue when tensorboard is correctly installed. That said, maybe we should look at not having this dependency 🤔

Would you mind posting a pip freeze of your environment that you're using to run yolo?

Based on the screenshot of you package versions, it does seem like tensorboard is not installed there. We depend on that, because every scalar logged to tensorboard is captured in ClearML too. My guess would be that maybe you installed tensorboard in e.g. the wrong virtualenv.

However, you do say you tested it with Tensorboard and even then it didn't work. In that case, are the scalars correctly logged to tensorboard? You should be able to easily check this by doing a run, and then launching...

I'm still struggling to reproduce the issue. Trying on my own PC locally as well as on google colab yields nothing.

The fact that you do get tensorboard logs, but none of them are captured by ClearML means there might be something wrong with our tensorboard bindings, but it's hard to pinpoint exactly what if I can't get it to fail like yours 😅 Let me try and instal exactly your environment using your packages above. Which python version are you using?

I'll update you once I have more!

The scheduler just downloads a dataset using the ID right? So if you don't upload a new dataset, the scheduler is just downloading the dataset from the last known ID then. I don't really see how that could lead to a new dataset with it's own ID as the parent. Would you mind explaining your setup in a little more detail? 🙂

That's a good idea! I think the YOLO models would be a great fit for a tutorial/example like this. We can add it to our internal list of TODOs, or if you want, you could take a stab at it and we'll try to support you through it 🙂 It might take some engineering though! Serving is never drag and drop 🙂

That said, I think it should be quite easy to do since YOLOv8 supports exporting to tensorrt format, which is native to Triton serving which underlies ClearML serving. So the process shoul...

How large are the datasets? To learn more you can always try to run something like line_profiler/kerprof, to get exactly how long a specific python line takes. How fast/stable is your internet?

Just to be sure I understand you correctly: you're saving/dumping an sklearn model in the clearml experiment manager, then want to serve it using clearml serving, but you do not wish to specify the model input and ouput shapes in the CLI?

Hi Oriel!

If you want to only serve an if-else model, why do you want to use clearml-serving for that? What do you mean by "online featurer"?

Thanks! I'm checking now, but might take a little (meeting in between)

If you didn't use git, then clearML saves your .py script completely in the uncommited changes section like you say. You should be able to just copy paste it to get the code. In what format are your uncommited changes logged? Can you paste a screenshot or paste the contents of uncommitted changes ?

When I run the example this way, everything seems to be working.

That looks like a bug, would you mind copy pasting this into a github issue? 🙂 AgitatedDove14 is there something else this could be?

Here is an example of deploying an sklearn model using ClearML serving.

However, please note that sklearn-like models don't have input and output shapes in the same sense as deep learning models have. Setting the I/O shapes using the CLI is usually meant for GPU-based deep learning models that need to know the sizes for better GPU allocation. In the case of sklearn on CPU, all you have to do is set up your preprocess...

That might make things harder indeed 🙂 It does explain some things

Check your agent logs (through clearml console tab) and check if there isn't any error thrown.

What is probably happening is that your agent tries to upload the model but fails due to some kind of networking/firewall/port issue. For example: make sure you host your self-hosted server on 0.0.0.0 host so it's able to accept external connections other than localhost