Hi NastyFox63
What do you mean not all of them are shown?
Do they have diff series/titles, are they plots or scalars ? How are you reporting them ?

good questions 🙂
they are plots. they have unique titles. i’m using the auto-logging mechanism—so set up the task, then plt.show() no more than 114 plots are shown in the plots tab.

since it’s probably relevant—i have to use the Agg backend since the machine is headless

using clearML 0.17.5

i’ve just verified that they’re all writen to /opt/clearml/data/fileserver/[PROJECT_NAME]/[DESCRIPTION]/metrics

running my own clearml server with a vanilla config (obtained from github), except i have one fixed user

Hi NastyFox63
This seems like most of the reports are converted to pngs (which is what the automagic does if it fails to convert the matplotlib into interactive plot).

no more than 114 plots are shown in the plots tab.

Are you saying we have 114 limit on plots ?
Is this true for "full screen" mode (i.e. not in the experiments table but switch to full detailed view)

yep, that’s what i’m seeing, they’re all PNGs in that folder.

yes, i see no more than 114 plots in the list on the left side in full screen mode—just checked and the behavior exists on safari and chrome

yes, i see no more than 114 plots in the list on the left side in full screen mode—just checked and the behavior exists on safari and chrome

Let me check with front-end guys 🙂

wondering if there has been an update on this?

Thanks for checking NastyFox63
I double checked with both front/backend , there should not be any limit...
Could you maybe provide a toy demo to reproduce the issue ?

https://demoapp.demo.clear.ml/projects/5ded4ded448a4e1f89460d9a8e675cc3/experiments/a3844407ae494ab2a88df5324d371346/output/metrics/plots

the items at the bottom of the list have dropped off—there’s no 2D hist 9 , or 2D hist 81 , etc.

Okay, this is odd the request returned exactly 100 out 100.
It seems not all of them were reported?!
Could you post the toy code, I'll check what's going on.

should be posted in the “uncommitted changes” section 🙂

if you’re able to check the data store, folders for all 120 plots will be on disk.

Let me rerun the code and check

I think I found something, let me test my theory

Okay there is some odd stuff going on in the backend, I'll check with backend guys tomorrow and update 🙂

good luck! thanks for looking into it 🙂

don’t want to pester, but i am curious—did they have some thoughts on what was happening? should i make a feature request somewhere?

Hi NastyFox63 yes I think the problem was found (actually backend side).
It will be solved in the upcoming release (due after this weekend 🙂 )

great news! thank you! when there’s a new release, i need to docker-compose build && docker-compose up to get the latest?

NastyFox63 ask SuccessfulKoala55 tomorrow, I think there is a way to change the default settings even with the current version.
(I.e. increase the default 100 entries limit)

thank you, will do.

Hi NastyFox63 ,
You can set the ClearML Server's services.events.events_retrieval.max_metrics_count and services.events.events_retrieval.max_variants_count settings.
Currently, these values are 100 and 100 respectively - this is why you only get 100 plots since you're probably using more than 100 variants (or series as its called when reporting a plot).
The only limitation is that the server cannot fetch more than 10,000 results when querying the ES backend for plots, so theoretically, even if you specify 1000 and 1000 respectively, you should be fine as long as you don't have more than a total of 10,000 plots reported for the given iteration.
Assuming you're using docker-compose , you can set up these configuration values using environment variables in the apiserver's env section, like so:
env: CLEARML__SERVICES__EVENTS__EVENTS_RETRIEVAL__MAX_METRICS_COUNT: 1000 CLEARML__SERVICES__EVENTS__EVENTS_RETRIEVAL__MAX_VARIANTS_COUNT: 1000You can also configure these using a configuration file, of course 🙂

great, thank you very much for this info!