How are the metrics being reported? directly via the Logger module or via automatical logging of some framework? Also how are iterations reported
Hi StickyShrimp60 , do you have any code that can reproduce this behavior?
Hi CostlyOstrich36 - unfortunately not something that I can easily share I am afraid. I am using the hosted solution btw and I really believe I am just logging way too often, I think about once a second for some 12 hours..
So early/mid experiment run everything aligned fine?
Yes, early in the experiment everything seems fine, then updating begins lagging and coming in chucks before not updating at all anymore
I am logging directly via the Logger module using the report_scalar method, adding iterations parameter in every call. In my recent runs I am only logging every 30s and this does seem to remove the issue do not see the issue