Hi all, I'm getting this "ERROR - Action failed <500/100: events.add_batch/v1.0" in my task log, can any one help diagnose \ solve this issue? what does it actually mean?
full line example:
2021-10-01 09:14:06,442 - clearml.Metrics - ERROR - Action failed <500/100: events.add_batch/v1.0 (General data error: err=('8 document(s) failed to index.', [{'index': {'_index': 'events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b', '_type': '_doc', '_id': '5da12fec94ed65c3d2d31e2422cc2f47', 'status': 429,..., extra_info=rejected execution of processing of [2643959][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b][0]] containing [8] requests and a refresh, target allocation id: 9k8zzYb7RvqHi47vrBVg8Q, primary term: 22 on EsThreadPoolExecutor[name = clearml/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@35da46b7[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 849977]])>