Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Have An Experiment That Generates Many Plots, But Not All Of Them Show Up In The “Plots” Section Of The Experiment Results. I Thought I Read Somewhere About A Limit On The Number Of Plots That Would Be Shown In That Section, But I Couldn’T Find It In Th

I have an experiment that generates many plots, but not all of them show up in the “plots” section of the experiment results. I thought i read somewhere about a limit on the number of plots that would be shown in that section, but I couldn’t find it in the slack history. Does anybody have information on this?

  
  
Posted 3 years ago
Votes Newest

Answers 27


Hi NastyFox63
What do you mean not all of them are shown?
Do they have diff series/titles, are they plots or scalars ? How are you reporting them ?

  
  
Posted 3 years ago

good questions 🙂
they are plots. they have unique titles. i’m using the auto-logging mechanism—so set up the task, then plt.show() no more than 114 plots are shown in the plots tab.

  
  
Posted 3 years ago

since it’s probably relevant—i have to use the Agg backend since the machine is headless

  
  
Posted 3 years ago

using clearML 0.17.5

  
  
Posted 3 years ago

i’ve just verified that they’re all writen to /opt/clearml/data/fileserver/[PROJECT_NAME]/[DESCRIPTION]/metrics

  
  
Posted 3 years ago

running my own clearml server with a vanilla config (obtained from github), except i have one fixed user

  
  
Posted 3 years ago

Hi NastyFox63
This seems like most of the reports are converted to pngs (which is what the automagic does if it fails to convert the matplotlib into interactive plot).

no more than 114 plots are shown in the plots tab.

Are you saying we have 114 limit on plots ?
Is this true for "full screen" mode (i.e. not in the experiments table but switch to full detailed view)

  
  
Posted 3 years ago

yep, that’s what i’m seeing, they’re all PNGs in that folder.

yes, i see no more than 114 plots in the list on the left side in full screen mode—just checked and the behavior exists on safari and chrome

  
  
Posted 3 years ago

yes, i see no more than 114 plots in the list on the left side in full screen mode—just checked and the behavior exists on safari and chrome

Let me check with front-end guys 🙂

  
  
Posted 3 years ago

wondering if there has been an update on this?

  
  
Posted 3 years ago

Thanks for checking NastyFox63
I double checked with both front/backend , there should not be any limit...
Could you maybe provide a toy demo to reproduce the issue ?

  
  
Posted 3 years ago

the items at the bottom of the list have dropped off—there’s no 2D hist 9 , or 2D hist 81 , etc.

  
  
Posted 3 years ago

Okay, this is odd the request returned exactly 100 out 100.
It seems not all of them were reported?!
Could you post the toy code, I'll check what's going on.

  
  
Posted 3 years ago

should be posted in the “uncommitted changes” section 🙂

  
  
Posted 3 years ago

if you’re able to check the data store, folders for all 120 plots will be on disk.

  
  
Posted 3 years ago

Let me rerun the code and check

  
  
Posted 3 years ago

I think I found something, let me test my theory

  
  
Posted 3 years ago

Okay there is some odd stuff going on in the backend, I'll check with backend guys tomorrow and update 🙂

  
  
Posted 3 years ago

good luck! thanks for looking into it 🙂

  
  
Posted 3 years ago

don’t want to pester, but i am curious—did they have some thoughts on what was happening? should i make a feature request somewhere?

  
  
Posted 3 years ago

Hi NastyFox63 yes I think the problem was found (actually backend side).
It will be solved in the upcoming release (due after this weekend 🙂 )

  
  
Posted 3 years ago

great news! thank you! when there’s a new release, i need to docker-compose build && docker-compose up to get the latest?

  
  
Posted 3 years ago

NastyFox63 ask SuccessfulKoala55 tomorrow, I think there is a way to change the default settings even with the current version.
(I.e. increase the default 100 entries limit)

  
  
Posted 3 years ago

thank you, will do.

  
  
Posted 3 years ago

Hi NastyFox63 ,
You can set the ClearML Server's services.events.events_retrieval.max_metrics_count and services.events.events_retrieval.max_variants_count settings.
Currently, these values are 100 and 100 respectively - this is why you only get 100 plots since you're probably using more than 100 variants (or series as its called when reporting a plot).
The only limitation is that the server cannot fetch more than 10,000 results when querying the ES backend for plots, so theoretically, even if you specify 1000 and 1000 respectively, you should be fine as long as you don't have more than a total of 10,000 plots reported for the given iteration.
Assuming you're using docker-compose , you can set up these configuration values using environment variables in the apiserver's env section, like so:
env: CLEARML__SERVICES__EVENTS__EVENTS_RETRIEVAL__MAX_METRICS_COUNT: 1000 CLEARML__SERVICES__EVENTS__EVENTS_RETRIEVAL__MAX_VARIANTS_COUNT: 1000You can also configure these using a configuration file, of course 🙂

  
  
Posted 3 years ago

great, thank you very much for this info!

  
  
Posted 3 years ago