looking into the output folder of catboost, I see 3 types of metrics outputs:
tfevents (can be read by tensorboard) catboost_training.json (custom (?) format). Is read here to be shown as an ipython widget: https://github.com/catboost/catboost/blob/c2a6ed0cb85869a73a13d08bf8df8d17320f8215/catboost/python-package/catboost/widget/ipythonwidget.py#L93 learn_error.tsv, test_error.tsv, time_left.tsv which have the same data as json. Apparently they are to be used with this stale metrics viewer project: https://github.com/catboost/catboost-viewer
nope, catboost docs offer to manually run tensorboard against the output folder https://catboost.ai/docs/features/visualization_tensorboard.html
Actually that is less interesting, as it is quite straight forward
Hmm I think everything is generated inside the c++ library code, and python is just an external interface. That means there is no was to collect the metrics as they are created (i.e. inside the c++ code), which means the only was to collect them is to actively analyze/read the tfrecord created by catboost š
Is there a python code that does that (reads the tfrecords it creates) ?
it certainly does not use tensorboard python lib
Hmm, yes I assume this is why the automagic is not working š
Does it have a pythonic interface form the metrics ?
Yes, but as you mentioned everything is created inside the lib, which means the python is not able to intercept the metrics so that clearml can send them to the backend.
Although it is only for model tracking, autologging is yet to be implemented there
Wanted to check if MLFlow supports catboost. Apparently, it does. Pull request was merged 16 hours ago. Nice timing š
as I understand, it uses tensorboard from C++ code
it certainly does not use tensorboard python lib
Hi FiercePenguin76
Is catboost actually using TB or is it just writing to .tfevent on its own ?
I guess this is the one https://catboost.ai/docs/concepts/python-reference_catboostipythonwidget.html