Hey There Have The Following Issue After Upgrading Server And Trains To 0.16:

Answered

hey there

Have the following issue after upgrading server and trains to 0.16:
Error 100 : General data error (TransportError(503, 'search_phase_execution_exception', 'Trying to create too many buckets. Must be less than or equal to: [10000] but was [11633]. This limit can be set by changing the [search.max_buckets] cluster level setting.'))Error appears when checking scalar plots. Randomly appeared after training for a while (it was ok for e.g. first epoch).

This seems to be coming from ES: https://discuss.elastic.co/t/search-max-buckets-limit-error-on-7-0-1/179989

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					ElegantKangaroo44
				
					0
					 × 1

Votes Newest

Answers 34

Might be some other issue related to loading plots from elastic. Can you show the trains-apiserver log again after you received the error - there should be some more information there

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

SubstantialBaldeagle49 This should collect the logs: 'sudo docker logs trains-apiserver >& apiserver.logs'

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					AppetizingMouse58
				
					0

Sure, you delete it with the following command:
curl -XDELETE " http://localhost:9200/events-plot-d1bd92a3b039400cbafc60a7a5b1e52b "
Once deleted it will be automatically recreated by the api server and should see the plots from the new tasks that you run afterwards

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					AppetizingMouse58
				
					0

AppetizingMouse58 Ok, this is the full log, here seems a error:

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					SubstantialBaldeagle49
				
					0
					 × 1

Show more results

Write your answer

25K Views

34 Answers

4 years ago

7 months ago