Hi

I just updated our server to the latest version, but it seems to have broken all our running experiments.
Scalars is totally down, I just get this error when going to the Scalars tab:

Error 100 : General data error (ApiError(503, 'search_phase_execution_exception', '[clearml][172.20.0.4:9300][indices:data/read/search[phase/query]]'))

Also our experiments log these errors:

2024-09-05 04:54:16,047 - clearml.Metrics - ERROR - Action failed <401/30: events.add_batch/v1.0 (Invalid token (invalid jwt token): reason=Signature verification failed)>
2024-09-05 04:55:17,836 - clearml.Metrics - ERROR - Action failed <500/100: events.add_batch/v1.0 (General data error: err=500 document(s) failed to index., extra_info=[events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b][0]] containing [500] requests and a refresh])>

and I'm at a loss as to what they mean. Must be related to the server restart.

Help?