Hi!
We use ClearML self-hosted on a k8s cluster. Great work btw 😉
We are currently experiencing a problem with the debug samples of tasks. They are not shown when going to the tab in the detail view of the task. In the logs of the Apiserver we see that there is an AuthorizationException 403 from Elasticsearch. My question is not regarding the exception itself, I think we can work this out but more on the general coupling between ClearML and Elasticsearch.
Why is the debug sample at all coupled with Elasticsearch? For me it is some sort of file I would have suspected to be stored on the Fileserver and things like the console output of a task would be stored on Elasticsearch. Why can then problems with Elasticsearch lead to problems with showing debug samples?
And if there is this tight connection to Elasticsearch: Would you recommend to have a dedicated Elasticsearch instance for ClearML? We are currently using a more global Elasticsearch instance where other systems are also logging to and problems there may directly impact ClearML if I see it right?
Hi @<1649221394904387584:profile|RattySparrow90> , events and console logs are logged to elastic so they can be fetched. Debug samples are also events so they are saved to elastic (A link to the debug is saved in the event itself).
It is suggested to keep a dedicated elastic to ClearML
@<1523701070390366208:profile|CostlyOstrich36> Thanks for the clarification 👍
