since it’s probably relevant—i have to use the Agg
backend since the machine is headless
wondering if there has been an update on this?
should be posted in the “uncommitted changes” section 🙂
Okay there is some odd stuff going on in the backend, I'll check with backend guys tomorrow and update 🙂
Okay, this is odd the request returned exactly 100 out 100.
It seems not all of them were reported?!
Could you post the toy code, I'll check what's going on.
running my own clearml
server with a vanilla config (obtained from github), except i have one fixed user
yes, i see no more than 114 plots in the list on the left side in full screen mode—just checked and the behavior exists on safari and chrome
Let me check with front-end guys 🙂
i’ve just verified that they’re all writen to /opt/clearml/data/fileserver/[PROJECT_NAME]/[DESCRIPTION]/metrics
good questions 🙂
they are plots. they have unique titles. i’m using the auto-logging mechanism—so set up the task, then plt.show()
no more than 114 plots are shown in the plots tab.
Hi NastyFox63
What do you mean not all of them are shown?
Do they have diff series/titles, are they plots or scalars ? How are you reporting them ?
yep, that’s what i’m seeing, they’re all PNGs in that folder.
yes, i see no more than 114 plots in the list on the left side in full screen mode—just checked and the behavior exists on safari and chrome
Hi NastyFox63
This seems like most of the reports are converted to pngs (which is what the automagic does if it fails to convert the matplotlib into interactive plot).
no more than 114 plots are shown in the plots tab.
Are you saying we have 114 limit on plots ?
Is this true for "full screen" mode (i.e. not in the experiments table but switch to full detailed view)
don’t want to pester, but i am curious—did they have some thoughts on what was happening? should i make a feature request somewhere?
Hi NastyFox63 ,
You can set the ClearML Server's services.events.events_retrieval.max_metrics_count
and services.events.events_retrieval.max_variants_count
settings.
Currently, these values are 100 and 100 respectively - this is why you only get 100 plots since you're probably using more than 100 variants (or series
as its called when reporting a plot).
The only limitation is that the server cannot fetch more than 10,000 results when querying the ES backend for plots, so theoretically, even if you specify 1000 and 1000 respectively, you should be fine as long as you don't have more than a total of 10,000 plots reported for the given iteration.
Assuming you're using docker-compose , you can set up these configuration values using environment variables in the apiserver's env section, like so:env: CLEARML__SERVICES__EVENTS__EVENTS_RETRIEVAL__MAX_METRICS_COUNT: 1000 CLEARML__SERVICES__EVENTS__EVENTS_RETRIEVAL__MAX_VARIANTS_COUNT: 1000
You can also configure these using a configuration file, of course 🙂
if you’re able to check the data store, folders for all 120 plots will be on disk.
great news! thank you! when there’s a new release, i need to docker-compose build && docker-compose up
to get the latest?
Thanks for checking NastyFox63
I double checked with both front/backend , there should not be any limit...
Could you maybe provide a toy demo to reproduce the issue ?
NastyFox63 ask SuccessfulKoala55 tomorrow, I think there is a way to change the default settings even with the current version.
(I.e. increase the default 100 entries limit)
I think I found something, let me test my theory
the items at the bottom of the list have dropped off—there’s no 2D hist 9
, or 2D hist 81
, etc.
Hi NastyFox63 yes I think the problem was found (actually backend side).
It will be solved in the upcoming release (due after this weekend 🙂 )