Hello Everyone, We Have Clearml (Free Version) Deployed On Gke. We Are Using Agents To Work With Gpu Nodes On The Gke. When We Start A Task On Gpu Worker, The Scalars Show'S That The Task Is Trying To Use The Entire Gpu'S On The Node (We Have 8 ) Even Th

Answered

Hello everyone,
We have ClearML (free version) deployed on GKE.
We are using agents to work with GPU nodes on the GKE.

When we start a task on GPU worker, the scalars show's that the task is trying to use the entire GPU's on the node (we have 8 ) even though we limit the worker to use only 1 gpu at the time so we can have 8 runs in parallel.

Is there any resource or config we need to set in order to inform ClearML that it should only refer to the GPU's the worker has requested?

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					JitteryCrab82
				
					0
					 × 1

Votes Newest

Answers 2

not sure about that in special, but take a look at the doc config: None
sometimes is difficult to understand the priority of the configs ( ENV_CONFIG, server yml, agent yml, etc.. ) because you can set a config by the agent config but the ENV is overwriting the config.

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					ThoughtlessTiger0
				
					0

It was related to the deployment having privileged permission enabled, after disabling it was fine.

Thanks

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					JitteryCrab82
				
					0
					 × 1

Write your answer

357 Views

2 Answers

2 months ago

Hey All, We Have Clearml Running On Our K8S On Prem With 4 Worker Nodes For Clearml-Agents And One Node For Clearml-Server. We Would Like To Start Using Clearml Datasets And Running Pipelineg Training On It. The Datasets Might Be Very Large, About 20G. I

Hey, I'M Working With The Clearml Aws Autoscaler (For Quite Some Time) And Suddenly We Encountered An Issue With Scaling Gpu Machines That Torch Inside The Task Doesn'T Recognize The Gpu Sporadically. If We Restart The Task It Works Just Fine... I Have A

Hello All, I Am New To Clearml, Need A Clarification. I Would Like To Enable Gpu-As-A-Service With Secure Multi-Tenancy And Real-Time Billing Per Tenant, Is This Feature Available In Open Source? Or Do We Need To Purchase The Enterprise Access For Multi-T

Hi, We Are Storing Lots And Lots Of Scalars On Worker Resource And Queue Stats. This Totals To About 29 Gb. Is There Any Way Of Limiting The Amount Of Stats Being Logged? We Are Not Very Interested In The Worker Stats And 11G For Just Monitoring The Queue

Hi All, We Have Clearml-Server Running On A Kube Pod, And Then A Gpu Server Running The Clearml-Agent Which We Use To Queue Jobs. For Some Reason, Our Kube Pod Restarted (We'Re Looking Into Why), But In The Process Of This Happening All Jobs On The Worke

Hey, How Can We Control The Pod Of Pipelinecontroller Not Use The Gpu? According To The Documentation, The

Hi :slightly_smiling_face: we have a clearml pipeline that has a step that runs a multi gpu training (with hugginface), we need to invoke it with `accelerate launch` so we use `subprocess.run` inside the step but it hangs when finished. Is this the righ

Hi, We Have An Agent Running Inside A Nvidia Official Container. The Agent Seems To See The Gpu Driver But The Gpu Count Is 0 When I Join That Container,

Hi Everyone!! Is Clearml Multi Gpu Support. I Have 2 Machines. Each Machine Have 2 A100 Gpu. Do We Have Any Option To Run Workload In These Gpu. Same Time Like Virtually. Do We Have The Support??