Hello All, I Installed Self-Hosted Server And Queue(Cosumes 1 Gpu) On Kubernetes. I Have An Issue Regarding Gpu Monitoring. I Checked The Process Is Using Gpu In The Pod, But Gpu Usage Is Not Being Displayed On Workers & Queues Dashboard, Whereas Cpu Usag

Answered

Hello all,
I installed self-hosted server and queue(cosumes 1 gpu) on kubernetes.
I have an issue regarding gpu monitoring.
I checked the process is using gpu in the pod, but gpu usage is not being displayed on WORKERS & QUEUES Dashboard, whereas CPU usage is. what is wrong?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TartLeopard58
				
					0
					 × 1

Votes Newest

Answers 40

@<1523701087100473344:profile|SuccessfulKoala55> yes. It only occurs when running on the cloud. It’s fine when running on-premises.

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TartLeopard58
				
					0
					 × 1

it is working on on-premise machine(i can see gpu usage on WORKERS & QUEUES Dashboard). but it is not working on cloud pod

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TartLeopard58
				
					0
					 × 1

This is clearml-agent helm chart values.yaml file i used to install

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TartLeopard58
				
					0
					 × 1

Can you share it?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Hi again 😊 @<1523701087100473344:profile|SuccessfulKoala55> sure!

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TartLeopard58
				
					0
					 × 1

I set CLEARML_AGENT_UPDATE_VERSION=1.5.3rc2 ` in agentk8sglue.basePodTemplate.env as i mentioned

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TartLeopard58
				
					0
					 × 1

Oh, It’s not the issue with eks.. We had the same issue on an on-premise cluster too(clearml-agent is installed). Could it be because of clearml-agent installed?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TartLeopard58
				
					0
					 × 1

@<1523701087100473344:profile|SuccessfulKoala55> Okay..but how can i specify agent’s verison in helm chart?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TartLeopard58
				
					0
					 × 1

Yes

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

I tried using K8S_GLUE_POD_AGENT_INSTALL_ARGS=1.5.3rc2 instead of CLEARML_AGENT_UPDATE_VERSION=1.5.3rc2 , but it’s same. doesn’t read gpu usage.. 🥲

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TartLeopard58
				
					0
					 × 1

Show more results

Write your answer

174K Views

40 Answers

2 years ago