Hi, i am logging some scalars from a simple ML experiment, and im seing different curves logged with clearml depending on when i run the code. At the moment we are basically drawing some random numbers and logging their values, but the frequency between each measurement point on the clearml website, differs between runs. Some runs there are minutes between measurements and some there are seconds. We can see that the code reaches the logging api every second and would expect the curves on the website to have datapoints every 1 second instead of every minute. The curves with 1 minute delays between datapoints also have far lower variance than the curves which has datapoints every second, which suggests that clearml is averaging the logged scalars some times, and sometimes not. The code being run are identical just at different times. Is clearml averaging in its logging somewhere or is there something we are missing? Appreciate your help!
It seems that the longer it runs, and the more datapoints it accumulates, it starts averaging over iterations
Hi StormySeaturtle98 ,how are you reporting these numbers?
Hi SuccessfulKoala55 We wonder if this is intended behavior from clearml? That the longer the experiment runs, it start averaging over datapoints. We have had experiments where the variance in the datapoints were significant, and this "feature" meant that our variance would become smaller and smaller?
