Problem: Excessive Scalar Storage from TensorBoard Integration Causing Out-of-Memory on ClearML Server Hi team, We’ve run into a problem with ClearML ingesti...
one month ago
Hi @<1523701070390366208:profile|CostlyOstrich36> , thank you for your quick response and for confirming the current behavior.
We’ve already tried the approaches you mentioned—disabling the TensorBoard autologging and reducing the scalar logging frequency—which definitely help for new experiments going forward.
However, our main challenge is with existing (“legacy”) tasks that already logged hundreds of thousands of scalars per experiment. We have temporarily alleviated the problem by adding...