I subscribe to the problem of having large metrics without a tool for proper inspection what is it coming from.

I tried to export them to json and they don't take more than 50KB each, but maybe they take more memory internally?

Ballpark should be the same.

I'm already at 300MB of usage with just 15 tasks

Maybe it was not updated yet? meaning you had more and deleted? (I think this is updated asynchronously, with max of 24h)

I can definitely feel you!
(I think the implementation is not trivial, metrics data size is collected and stored as commutative value on the account, going over per Task is actually quite taxing for the backend, maybe it should be an async request ? like get me a list of the X largest Tasks? How would the UI present it? As fyi, keeping some sort of book keeping per task is not trivial either, hence the main issue)

I'm already at 300MB of usage with just 15 tasks

Wow, what do you have there? I would try to download the console logs and see what the size you are getting, this is the only thing that makes sense, wdyt?

BTW: to get the detailed size for scalars, maximize the plot (otherwise you are getting "subsampled" data)

Hi @<1523701087100473344:profile|SuccessfulKoala55> , I'm uploading some debug images by they are around 300KB each, and less than 10 per experiment. Also, aren't debug images counted as artifacts for the quota?

So the longest experiments I have takes ~800KB in logs. I have tens of plotly plots logged manually, how are they stored internally? I tried to export them to json and they don't take more than 50KB each, but maybe they take more memory internally?

Hi @<1523701205467926528:profile|AgitatedDove14> , I already tried to check manually in the web UI for some anomalous file, i.e. by downloading the log files or exporting the metrics plots, but I couldn't find anything that takes more than 100KB, and I'm already at 300MB of usage with just 15 tasks. It's not possible to get more info using some python APIs?

UI for some anomalous file,

Notice the metrics are not files/artifacts, just scalars/plots/console

Also, very large git diffs and/or connected configuration might also grow fairly large

Would just having some python API be an option? It would be more than enough to check what is causing this, and it would be called infrequently

Like get the tasks that uses the most metrics API?

I deleted a few experiments, but they had the same kind of plots and metrics. so I don't think they would release much space

Hi @<1570220858075516928:profile|SlipperySheep79>
I think this is more complicated than one would expect. But as a rule of thumb, console logs and metrics are the main ones. I hope it helps? Maybe sort by number of iterations in the experiment table ?

BTW: probable better to ask in channel

Yes, or even just something like task.get_size()

I have some git diffs logged but they are very small. For the configurations I saw that the datasets tasks have a fairly large "Dataset Content" config (~2MB), but I only have 5 dataset tasks

Hi @<1570220858075516928:profile|SlipperySheep79> , are you by any chance uploading large debug images?