Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Guys! What Can Be Wrong Here And How I Can It Fix: Trains.Log - Warning - Failed Logging Task To Backend (2 Lines, <500/100: Events.Add_Batch/V1.0 (General Data Error: Err=('2 Document(S) Failed To Index.', [{'Index': {'_Index': 'Events-Log-D1Bd92A3B0

Hi guys! What can be wrong here and how I can it fix:

trains.log - WARNING - failed logging task to backend (2 lines, <500/100: events.add_batch/v1.0 (General data error: err=('2 document(s) failed to index.', [{'index': {'_index': 'events-log-d1bd92a3b039400cbafc60a7a5b1e52b', '_type': '_doc', '_id': 'd48015232deb41b88d2722a6b8426d56', 'status': 403, 'error': {'type': 'cluster_block_exception', 'reason': 'index [events-log-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];'}, 'data': {'timestamp': 1599124998933, 'type': 'log', 'task': '4e1210eed9ba4f9e91573701b9bfea4a', 'level': 'info', 'worker': ' http://koala03.ftc.ru ', 'msg': 'TRAINS Task: overwriting (reusing) task id=4e1210eed9ba4f9e91573701b9bfea4a\n======> WARNING! UNCOMMITTED CHANGES IN REPOSITORY https://git.ftc.ru/ml/projects/rec-registers-ru-mail/modelling/meter_classification.git <======\nTRAINS results page: http://koala03.ftc.ru:8080/projects/2560f347bcf54c668312a6b30129cc77/experiments/4e1210eed9ba4f9e91573701b9bfea4a/output/log ', '@timestamp': '2020-09-03T09:23:20.947Z', 'metric': '', 'variant': ''}}}, {'index': {'_index': 'events-log-d1bd92a3b039400cbafc60a7a5b1e52b', '_type': '_doc', '_id': '2151a84aac6f45789eddc869549fc2cd', 'status': 403, 'error': {'type': 'cluster_block_exception', 'reason': 'index [events-log-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];'}, 'data': {'timestamp': 1599124999939, 'type': 'log', 'task': '4e1210eed9ba4f9e91573701b9bfea4a', 'level': 'info', 'worker': ' http://koala03.ftc.ru ', 'msg': 'TRAINS new version available: upgrade to v0.16.1 is recommended!', '@timestamp': '2020-09-03T09:23:20.947Z', 'metric': '', 'variant': ''}}}]), extra_info=index [events-log-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];)>)

trains.Metrics - ERROR - Action failed <500/100: events.add_batch/v1.0 (General data error: err=('1 document(s) failed to index.', [{'index': {'_index': 'events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b', '_type': '_doc', '_id': '78c1e7fd0f771998d287662fd74a107d', 'status': 403, 'error': {'type': 'cluster_block_exception', 'reason': 'index [events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];'}, 'data': {'timestamp': 1599125000275, 'type': 'training_debug_image', 'task': '4e1210eed9ba4f9e91573701b9bfea4a', 'iter': 0, 'metric': 'F1-Full', 'variant': 'text_summary', 'key': 'REGINA_Meter_classification/MixUp_test.4e1210eed9ba4f9e91573701b9bfea4a/metrics/F1-Full/text_summary/F1-Full_text_summary_00000000.txt', 'url': ' http://koala03.ftc.ru:8082/REGINA_Meter_classification/MixUp_test.4e1210eed9ba4f9e91573701b9bfea4a/metrics/F1-Full/text_summary/F1-Full_text_summary_00000000.txt ', '@timestamp': '2020-09-03T09:23:20.991Z', 'worker': ' http://koala03.ftc.ru '}}}]), extra_info=index [events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];)>
trains.Metrics - ERROR - Failed reporting metrics: <500/100: events.add_batch/v1.0 (General data error: err=('1 document(s) failed to index.', [{'index': {'_index': 'events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b', '_type': '_doc', '_id': '78c1e7fd0f771998d287662fd74a107d', 'status': 403, 'error': {'type': 'cluster_block_exception', 'reason': 'index [events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];'}, 'data': {'timestamp': 1599125000275, 'type': 'training_debug_image', 'task': '4e1210eed9ba4f9e91573701b9bfea4a', 'iter': 0, 'metric': 'F1-Full', 'variant': 'text_summary', 'key': 'REGINA_Meter_classification/MixUp_test.4e1210eed9ba4f9e91573701b9bfea4a/metrics/F1-Full/text_summary/F1-Full_text_summary_00000000.txt', 'url': ' http://koala03.ftc.ru:8082/REGINA_Meter_classification/MixUp_test.4e1210eed9ba4f9e91573701b9bfea4a/metrics/F1-Full/text_summary/F1-Full_text_summary_00000000.txt ', '@timestamp': '2020-09-03T09:23:20.991Z', 'worker': ' http://koala03.ftc.ru '}}}]), extra_info=index [events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];)>

trains.log - WARNING - failed logging task to backend (1 lines, <500/100: events.add_batch/v1.0 (General data error: err=('1 document(s) failed to index.', [{'index': {'_index': 'events-log-d1bd92a3b039400cbafc60a7a5b1e52b', '_type': '_doc', '_id': 'c6abd93b93bc4a52803562fdafebec30', 'status': 403, 'error': {'type': 'cluster_block_exception', 'reason': 'index [events-log-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];'}, 'data': {'timestamp': 1599125000999, 'type': 'log', 'task': '4e1210eed9ba4f9e91573701b9bfea4a', 'level': 'info', 'worker': ' http://koala03.ftc.ru ', 'msg': "2020-09-03 16:23:20,998 - trains.Metrics - ERROR - Action failed <500/100: events.add_batch/v1.0 (General data error: err=('1 document(s) failed to index.', [{'index': {'_index': 'events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b', '_type': '_doc', '_id': '78c1e7fd0f771998d287662fd74a107d', 'status': 403, 'error': {'type': 'cluster_block_exception', 'reason': 'index [events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];'}, 'data': {'timestamp': 1599125000275, 'type': 'training_debug_image', 'task': '4e1210eed9ba4f9e91573701b9bfea4a', 'iter': 0, 'metric': 'F1-Full', 'variant': 'text_summary', 'key': 'REGINA_Meter_classification/MixUp_test.4e1210eed9ba4f9e91573701b9bfea4a/metrics/F1-Full/text_summary/F1-Full_text_summary_00000000.txt', 'url': ' http://koala03.ftc.ru:8082/REGINA_Meter_classification/MixUp_test.4e1210eed9ba4f9e91573701b9bfea4a/metrics/F1-Full/text_summary/F1-Full_text_summary_00000000.txt ', '@timestamp': '2020-09-03T09:23:20.991Z', 'worker': ' http://koala03.ftc.ru '}}}]), extra_info=index [events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];)>\n2020-09-03 16:23:20,999 - trains.Metrics - ERROR - Failed reporting metrics: <500/100: events.add_batch/v1.0 (General data error: err=('1 document(s) failed to index.', [{'index': {'_index': 'events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b', '_type': '_doc', '_id': '78c1e7fd0f771998d287662fd74a107d', 'status': 403, 'error': {'type': 'cluster_block_exception', 'reason': 'index [events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];'}, 'data': {'timestamp': 1599125000275, 'type': 'training_debug_image', 'task': '4e1210eed9ba4f9e91573701b9bfea4a', 'iter': 0, 'metric': 'F1-Full', 'variant': 'text_summary', 'key': 'REGINA_Meter_classification/MixUp_test.4e1210eed9ba4f9e91573701b9bfea4a/metrics/F1-Full/text_summary/F1-Full_text_summary_00000000.txt', 'url': ' http://koala03.ftc.ru:8082/REGINA_Meter_classification/MixUp_test.4e1210eed9ba4f9e91573701b9bfea4a/metrics/F1-Full/text_summary/F1-Full_text_summary_00000000.txt ', '@timestamp': '2020-09-03T09:23:20.991Z', 'worker': ' http://koala03.ftc.ru '}}}]), extra_info=index [events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];)>", '@timestamp': '2020-09-03T09:23:22.946Z', 'metric': '', 'variant': ''}}}]), extra_info=index [events-log-d1bd92a3b039400cbafc60a7a5b1e52b] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];)>)

  
  
Posted 3 years ago
Votes Newest

Answers 2


Hi ItchyHippopotamus18 - your server machine disk storage is probably running low. See this GitHub Issue: https://github.com/allegroai/trains/issues/201

  
  
Posted 3 years ago

SuccessfulKoala55 Thanks you

  
  
Posted 3 years ago
519 Views
2 Answers
3 years ago
one year ago
Tags