Reputation
Badges 1
6 × Eureka!Hi @<1523701087100473344:profile|SuccessfulKoala55> We wonder if this is intended behavior from clearml? That the longer the experiment runs, it start averaging over datapoints. We have had experiments where the variance in the datapoints were significant, and this "feature" meant that our variance would become smaller and smaller?
Hi Eugen
Thanks for the response!
let me know if you need a running example, that will require some effort to produce.
It seems that running Task.init() outside in the main process before spawning multiple processes, and calling Task.current_task() inside the rank = 0 process works with no slowdown
It seems that the longer it runs, and the more datapoints it accumulates, it starts averaging over iterations
With the logger.report_scalar
It just puzzles me that if a single subprocess spawns a task, meaning that only one task is active, and that task is only accessed by the worker creating it, the massive slowdown happens