Hey, Everybody! I Am A New User Of The Clearml Service, And I Would Like To Ask You About Your Experience With Clearml Working With An Aws Virtual Machine. My Problem Is That When The Aws Virtual Machine Is Killed, My Pipelines And Scheduling Stop Working

Answered

Hey, everybody!
I am a new user of the ClearML service, and I would like to ask you about your experience with ClearML working with an AWS virtual machine. My problem is that when the AWS virtual machine is killed, my Pipelines and Scheduling stop working because of the killed ClearML agent, and I have to re-enter the app credentials and initialize the agent on a new startup. Someone may have encountered this problem and know how to work around it or fix it.
Thank you very much in advance!

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SpotlessOwl43
				
					0
					 × 1

Votes Newest

Answers 4

@<1523701205467926528:profile|AgitatedDove14> I am spinning it manually

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SpotlessOwl43
				
					0
					 × 1

Ohh then use the AWS autoscaler, basically it what you want, spin an EC2 and set an agent there, then if the EC2 goes down (for example if this is a spot), it will spin it up again automatically with the running Task on it.
wdyt?

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Okay, thanks, I’ll try

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SpotlessOwl43
				
					0
					 × 1

Hi @<1661904968040321024:profile|SpotlessOwl43>

My problem is that when the AWS virtual machine is killed, my Pipelines and Scheduling stop working because of the killed ClearML agent,

are you using the ClearML AWS autoscaler to spin that machine ? or are you spinning it manually ?

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Write your answer

622 Views

4 Answers

10 months ago

9 months ago