Hi Everyone, I Am Just Wondering Whether The Bugs Regarding The Deletion Of Tasks Is Fixed In The Current Version? E.G. This Happening When You Want To Delete A Lot Of Tasks.

Answered

Hi everyone,
I am just wondering whether the bugs regarding the deletion of tasks is fixed in the current version?
E.g. this happening when you want to delete a lot of tasks.
ERROR - Task deletion failed: Action failed <500/100: tasks.delete/v1.0 (General data error (TransportError(500, 'search_phase_execution_exception', 'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')))> (move_to_trash=False, force=True, task=8cb074988bec4f398096909957feaa16, delete_external_artifacts=True)

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

Votes Newest

Answers 6

Hi @<1523701868901961728:profile|ReassuredTiger98> , what version of the apiserver are you using?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					AppetizingMouse58
				
					0

curl -X PUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d'{"persistent" : {"search.max_open_scroll_context": 1000}}'

  				
Posted 
	one year ago

					More
				  		
  Report
		
					AppetizingMouse58
				
					0

Hi @<1523701868901961728:profile|ReassuredTiger98> , this is actually an issue with ES resources being exahusted. Does it only happen when you delete? Is this a self hosted server?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

@<1523701087100473344:profile|SuccessfulKoala55> Only when I delete on self-hosted.
@<1523712723274174464:profile|LazyFish41> WebApp: 1.10.0-357 • Server: 1.10.0-357 • API: 2.24

This has been happening with every version of clearml-server ever. Most probably there should be a queue in front of ES, so it does not process to many request at the same time?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

@<1523701868901961728:profile|ReassuredTiger98> Strange:( in 1.10 we already had the code for clearing ES scrolls created during the task deletion. I would recommend upgrading to the latest release v1.12.1 anyway. In addition you can instruct ES to allow more open scrolls like below. By default it is limited to 500.

  				
Posted 
	one year ago

					More
				  		
  Report
		
					AppetizingMouse58
				
					0

Thanks a lot. Will try it!

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

Write your answer

1K Views

6 Answers

one year ago