Hey, Everybody! I Am A New User Of The Clearml Service, And I Would Like To Ask You About Your Experience With Clearml Working With An Aws Virtual Machine. My Problem Is That When The Aws Virtual Machine Is Killed, My Pipelines And Scheduling Stop Working

Answered

Hey, everybody!
I am a new user of the ClearML service, and I would like to ask you about your experience with ClearML working with an AWS virtual machine. My problem is that when the AWS virtual machine is killed, my Pipelines and Scheduling stop working because of the killed ClearML agent, and I have to re-enter the app credentials and initialize the agent on a new startup. Someone may have encountered this problem and know how to work around it or fix it.
Thank you very much in advance!

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SpotlessOwl43
				
					0
					 × 1

Votes Newest

Answers 4

Hi @<1661904968040321024:profile|SpotlessOwl43>

My problem is that when the AWS virtual machine is killed, my Pipelines and Scheduling stop working because of the killed ClearML agent,

are you using the ClearML AWS autoscaler to spin that machine ? or are you spinning it manually ?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

@<1523701205467926528:profile|AgitatedDove14> I am spinning it manually

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SpotlessOwl43
				
					0
					 × 1

Ohh then use the AWS autoscaler, basically it what you want, spin an EC2 and set an agent there, then if the EC2 goes down (for example if this is a spot), it will spin it up again automatically with the running Task on it.
wdyt?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Okay, thanks, I’ll try

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SpotlessOwl43
				
					0
					 × 1

Write your answer

632 Views

4 Answers

10 months ago