Probably My Question Will Be About Something Other Than Core Concepts Of Clearml, But Something Connected I Think.... I Want To Schedule Tasks On Gpu And I Want Many Agents To Use Given Gpu At One Time. I Also Want To Avoid Situation When Gpu Will Be Out

Answered

Probably my question will be about something other than core concepts of ClearML, but something connected I think....

I want to schedule tasks on GPU and I want many agents to use given GPU at one time. I also want to avoid situation when GPU will be out of memory. But I also would like to use GPU as fully as possible. Do you have any tips on how to do that?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					RoundMosquito25
				
					0
					 × 1

Votes Newest

Answers 2

Hi RoundMosquito25
The main problem here is there is no way to know before running the Task how much memory it would need ... And without that parameter maximizing GPUs is quite challenging. wdyt?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Yes, it is a good reason 🙂
Do you maybe know a tool that measures that during execution (to avoid looking on nvidia-smi during all training)?

So, suppose, that a task T uses 27% of GPU, means, that we can spawn 3 agents on this GPU (suppose that we will give them only task T). Does it make sense?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					RoundMosquito25
				
					0
					 × 1

Write your answer

1K Views

2 Answers

2 years ago

one year ago