Unanswered
Hey! I Stumbled Upon Some Errors With My Workers Monitoring.
I Checked Logs In My K8S Pods For Apiserver And Elasticsearch And It Seems The Problem Is There. These Are The Logs:
Apiserver Logs
[2021-04-23 06:19:50,209] [9] [Error] [Trains.Service_Repo] Re
Hi again GreasyPenguin66 🙂
For some reason, it looks like the mapping for the Elastic index containing the worker (agents) statistics were not initialized correctly - this happens automatically when the ClearML server starts up. The server might not perform this auto-initialization in case it suspects the ES data as originating from an un-migrated pre-v16 Trains Server deployment (I'm not sure this is the case here)
99 Views
0
Answers
2 years ago
one year ago