Hi, I Recently Updated Clearml-Server To 1.7 And I Am Getting A Lot Of The Following Errors Since Today On Any Experiment (I Didn'T Had This Error Before):

Answered

Hi, I recently updated clearml-server to 1.7 and I am getting a lot of the following errors since today on any experiment (I didn't had this error before):
1665060540239 my-agent DEBUG 2022-10-06 12:48:59,573 - clearml.Metrics - ERROR - Action failed <500/100: events.add_batch/v1.0 (General data error: err=('1 document(s) failed to index.', [{'index': {'_index': 'events-plot-d1bd92a3b039400cbafc60a7a5b1e52b_new', '_type': '_doc', '_id': '5aa0d48b1215dfc69caf5fa89b9c4f6c', 'status': 400, 'error':..., extra_info=mapping set to strict, dynamic introduction of [source_urls] within [_doc] is not allowed)>Can it be due to the update? When was this source_urls field introduced? How can I add it?
clearml sdk: 1.7
clearml-agent: 1.4

EDIT: I think it is because of https://github.com/allegroai/clearml-server/compare/v1.6.0...v1.7.0#diff-77864ff70522bd69170842d9d3360710c5139467701d5d7bed167252228c2125R72-R81 in the clearml-server?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					JitteryCoyote63
				
					0
					 × 1

Votes Newest

Answers 7

Hi JitteryCoyote63 , can I assume you ran all DB migrations without issue?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

If Elastic isn't crashing then it should be good. Once I get a confirmation I'll update you 🙂

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

JitteryCoyote63 I think you're right, I see no other explanation...

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

This is the mapping of the faulty index:
{ "events-plot-d1bd92a3b039400cbafc60a7a5b1e52b_new" : { "mappings" : { "dynamic" : "strict", "properties" : { "@timestamp" : { "type" : "date" }, "iter" : { "type" : "long" }, "metric" : { "type" : "keyword" }, "plot_data" : { "type" : "binary" }, "plot_len" : { "type" : "long" }, "plot_str" : { "type" : "text", "index" : false }, "task" : { "type" : "keyword" }, "timestamp" : { "type" : "date" }, "type" : { "type" : "keyword" }, "value" : { "type" : "float" }, "variant" : { "type" : "keyword" }, "worker" : { "type" : "keyword" } } } } }

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					JitteryCoyote63
				
					0
					 × 1

Hi CostlyOstrich36 , there was no DB migration necessary since 1.6, right?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					JitteryCoyote63
				
					0
					 × 1

I think maybe you're right. Let me double check. I might be confusing it with the previous version

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

To be fully transparent, I did a manual reindexing of the whole ES DB one year ago after it run out of space, at that point I might have changed the mapping to strict, but I am not sure. Could you please confirm that the mapping is correct?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					JitteryCoyote63
				
					0
					 × 1

Write your answer

1K Views

7 Answers

2 years ago

one year ago