I Have A Training Script That I Ran On A Local Machine With Gpu (2070 Super) And It Works Normally (It Utilizes The Gpu), But When I'M Training It On

Answered

I have a training script that I ran on a local machine with GPU (2070 super) and it works normally (it utilizes the GPU), but when I'm training it on ClearML GPU Compute (3090) it doesn't use the GPU per attached image

I'm using lighting as the training framework, is there something in particular I need to configure, so it uses the GPU on ClearML GPU Compute

Python Env:

fastapi==0.88.0
albumentations==1.3.0
torchinfo==1.8.0
lightning==2.0.2
clearml==1.10.4
segmentation-models-pytorch==0.3.2
transformers==4.29.2
Pillow==9.2.0
funcy==2.0
argparse
scikit-multilearn
matplotlib

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CrabbySquid74
				
					0

Votes Newest

Answers 2

Hi @<1572757559406759936:profile|CrabbySquid74> , are you sure that the machine has all the drivers installed and is capable of running experiments regardless of ClearML?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

i am having same issue: None

  				
Posted 
	one year ago

					More
				  		
  Report
		
					TartLeopard58
				
					0
					 × 1

Write your answer

1K Views

2 Answers

one year ago

Hi :slightly_smiling_face: we have a clearml pipeline that has a step that runs a multi gpu training (with hugginface), we need to invoke it with `accelerate launch` so we use `subprocess.run` inside the step but it hangs when finished. Is this the righ

Just Want To Post It Here Before Raising A Github Issue: There Seems To Be A Regression Bug Since Clearml 1.13.0 Where Out Training In Gpu Is 2X Slower In Our Pipeline Based On

Is There A Way To Configure A Clearml-Agent So That It Shutdown The Server After It Has Been Idle For A Certain Time Period? We Are Using Gpu Resources From A Provider That Autoscaling Doesn'T Support (Such As Sagemaker Training Jobs).

Hi Everyone, I'M Using Torch.Distributed For Training On 2 Gpus. It Works, But Each Gpu Creates A New (Duplicated) Task, And I Prefer To Have Only One Clearml Experiment Running. I Looked Here

Hi, We Are Encountering An Increasing Number Of Cases Where It Takes Quite A While Before Actual Training (Gpu Utilisation) Can Be Done. After Observing, This Is What We Discovered. The Following Are The Steps And Bottlenecks.

Hey, I Run A Programm Without Allegro On The Gpu And It Works. Then I Run It With Allegro. But The Training Does Not Start. The Gpu Is Allocated But The Training Does Not Start. The Programm Is Stuck. I Am Using The Newest Allegro Version 1.0.2 How Can I

I Got An Interesting Question From My Devs. If They Wish To Do Distributed Training, Is Clearml K8S Glue Suitable For It? Local Multiple Gpu: Just A Matter Of Assigning More Than One Gpu In The Yaml File Sent To The K8S Glue. Question Is How To Make This