Is It Possible To Increase The Time A Task Takes To Idle Abort On A Managed Clearml Server? Some Long Running Tasks Without Active Logging Are Timing Out To Abort And Then Error When Artifact Attachment Occurs Since The Task State Is Aborted.

Answered

Is it possible to increase the time a task takes to idle abort on a managed clearml server? Some long running tasks without active logging are timing out to abort and then error when artifact attachment occurs since the task state is aborted.

  				
Posted 
	one year ago

					More
				  		
  Report
		
					PerplexedElk26
				
					0
					 × 1

Votes Newest

Answers 3

Well, if you use the latest agent version to run your tasks remotely, it actually does that for you 🙂

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Hi PerplexedElk26 , do you mean the timeout used by the server's non-responsive tasks monitor that marks tasks as aborted after 2 hours of not reporting anything?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Hey SuccessfulKoala55 , yup, can we increase that timeout length? I could work around that by having dummy task updates just to maintain the connection

  				
Posted 
	one year ago

					More
				  		
  Report
		
					PerplexedElk26
				
					0
					 × 1

Write your answer

542 Views

3 Answers

one year ago