running my own clearml server with a vanilla config (obtained from github), except i have one fixed user
Thanks for checking NastyFox63
I double checked with both front/backend , there should not be any limit...
Could you maybe provide a toy demo to reproduce the issue ?
if you’re able to check the data store, folders for all 120 plots will be on disk.
Hi NastyFox63 yes I think the problem was found (actually backend side).
It will be solved in the upcoming release (due after this weekend 🙂 )
I think I found something, let me test my theory
Hi NastyFox63 ,
You can set the ClearML Server's services.events.events_retrieval.max_metrics_count and services.events.events_retrieval.max_variants_count settings.
Currently, these values are 100 and 100 respectively - this is why you only get 100 plots since you're probably using more than 100 variants (or series as its called when reporting a plot).
The only limitation is that the server cannot fetch more than 10,000 results when querying the ES backend for plots, so theoretically, even if you specify 1000 and 1000 respectively, you should be fine as long as you don't have more than a total of 10,000 plots reported for the given iteration.
Assuming you're using docker-compose , you can set up these configuration values using environment variables in the apiserver's env section, like so:env: CLEARML__SERVICES__EVENTS__EVENTS_RETRIEVAL__MAX_METRICS_COUNT: 1000 CLEARML__SERVICES__EVENTS__EVENTS_RETRIEVAL__MAX_VARIANTS_COUNT: 1000You can also configure these using a configuration file, of course 🙂
NastyFox63 ask SuccessfulKoala55 tomorrow, I think there is a way to change the default settings even with the current version.
(I.e. increase the default 100 entries limit)
the items at the bottom of the list have dropped off—there’s no 2D hist 9 , or 2D hist 81 , etc.
since it’s probably relevant—i have to use the Agg backend since the machine is headless
great news! thank you! when there’s a new release, i need to docker-compose build && docker-compose up to get the latest?
don’t want to pester, but i am curious—did they have some thoughts on what was happening? should i make a feature request somewhere?
yep, that’s what i’m seeing, they’re all PNGs in that folder.
yes, i see no more than 114 plots in the list on the left side in full screen mode—just checked and the behavior exists on safari and chrome
Okay there is some odd stuff going on in the backend, I'll check with backend guys tomorrow and update 🙂
Hi NastyFox63
This seems like most of the reports are converted to pngs (which is what the automagic does if it fails to convert the matplotlib into interactive plot).
no more than 114 plots are shown in the plots tab.
Are you saying we have 114 limit on plots ?
Is this true for "full screen" mode (i.e. not in the experiments table but switch to full detailed view)
yes, i see no more than 114 plots in the list on the left side in full screen mode—just checked and the behavior exists on safari and chrome
Let me check with front-end guys 🙂
Hi NastyFox63
What do you mean not all of them are shown?
Do they have diff series/titles, are they plots or scalars ? How are you reporting them ?
should be posted in the “uncommitted changes” section 🙂
wondering if there has been an update on this?
Okay, this is odd the request returned exactly 100 out 100.
It seems not all of them were reported?!
Could you post the toy code, I'll check what's going on.
good questions 🙂
they are plots. they have unique titles. i’m using the auto-logging mechanism—so set up the task, then plt.show() no more than 114 plots are shown in the plots tab.
i’ve just verified that they’re all writen to /opt/clearml/data/fileserver/[PROJECT_NAME]/[DESCRIPTION]/metrics