Hey, I'M Working With The Clearml Aws Autoscaler (For Quite Some Time) And Suddenly We Encountered An Issue With Scaling Gpu Machines That Torch Inside The Task Doesn'T Recognize The Gpu Sporadically. If We Restart The Task It Works Just Fine... I Have A

Answered

Hey,
i'm working with the clearml AWS autoscaler (for quite some time) and suddenly we encountered an issue with scaling GPU machines that torch inside the task doesn't recognize the gpu sporadically. if we restart the task it works just fine... i have a feeling it might be related to initiating the task too soon

Help please

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					PricklyRaven28
				
					0
					 × 1

Votes Newest

Answers 2

Hey, it took me some to check out.

I added 20 retries to check gpu driver, it says it finds the driver, but still the task starts without gpu driver

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					PricklyRaven28
				
					0
					 × 1

Hi @<1523701949617147904:profile|PricklyRaven28> , I assume this is happening on the same instance? What if you put in like 20 sec sleep before or after the init call, does this behaviour reproduce?

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

476 Views

2 Answers

4 months ago

3 months ago