Hello,
I have been using clearml for my projects lately. I have set it up to log the training data for an RL training job. As of now it currently logs data as follows: 1 Million episodes and Rewards for 3 agents each
I'd have guessed that the storage requirement will be 4 bytes * 3 agents * 1M = 12Million bytes or around 12 mega byes. (I'd think it would be even lesser as data could be readily compressed) But clearml metrics showed around 600 MB usage.
I'd like to understand why this is the case, is there a way to find what exactly it is storing.
I tried downloading the data to examine what was being stored but couldn't find any way to do that from the clearml dashboard.
Hi @<1726772411946242048:profile|CynicalBlackbird36> , what you're looking at is the metrics storage, this is referring to all the console outputs, scalars, plots and debug samples.
This is saved in the backend of ClearML. There is no direct way to pull this but you can technically fetch all of this using the API.
wdyt?
