Hi @<1853245764742942720:profile|DepravedKoala88> , I don't think there is any downsampling when ingesting from Tensorboard. You can always turn off the autologging and only log what you want and downsample accordingly. Keep in mind that on one hand you should avoid bloat on the server and on the other have high enough granularity in your scalars.
What do you think?
Hi @<1523701070390366208:profile|CostlyOstrich36> , thank you for your quick response and for confirming the current behavior.
We’ve already tried the approaches you mentioned—disabling the TensorBoard autologging and reducing the scalar logging frequency—which definitely help for new experiments going forward.
However, our main challenge is with existing (“legacy”) tasks that already logged hundreds of thousands of scalars per experiment. We have temporarily alleviated the problem by adding more RAM. However, this isn't a sustainable solution
It would be extremely helpful to have a way to downsample or prune excessive scalars for past/existing tasks directly on the server. Is there any possibility that ClearML might implement such a feature or provide an admin tool/script for this purpose? This would be very valuable for maintenance, resource management, and scalability.
Thank you again!