Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Currently Try To Set Up A Way To Do Hyperparameter Optimization On 2 Machines. My Plan Would Be To Set Up A Clearml Worker On Each Of Those 2 Machines, With A Schedule To Pull New Task At Certain Times. Since We Are Still A Small Team, Those Machin

Hi,

I currently try to set up a way to do hyperparameter optimization on 2 machines. My plan would be to set up a ClearML Worker on each of those 2 machines, with a schedule to pull new task at certain times. Since we are still a small team, those machines are also used by other members of our team. My question is now: can I somehow check the available system resources of the machine and only pull a new task if, lets say the GPU utilization is below 10%? I get that I can set up a schedule for my workers, but it would be nice if I could also check if there are enough system resources available on the worker machines before pulling a task from the queue.

Thanks in advance!

  
  
Posted 4 months ago
Votes Newest

Answers 4


Hi @<1826066729852211200:profile|DullSwallow71> , I would suggest looking at it from another perspective. Check machine availability and only then push a job into a queue. You can see all the related information to usage in the 'Workers' screen in the 'Orchestration' tab.

Then you can either push jobs manually according to usage or write your own service to sample usage and push jobs accordingly.

  
  
Posted 4 months ago

Thanks for the reply, but I guess nothing ever came from it, right?

That's a bummer. I guess I have to hack something together myself then.

  
  
Posted 4 months ago

Thanks @<1523701070390366208:profile|CostlyOstrich36> for your suggestion. However I would like to automate this as much as possible and let it run outside of my working, so checking it manually would be my least favorite option.
The idea behind this would be to use the "downtime" of our machines to check for better hyperparameters, but because this hasn't the highest priority we would not kill any other process just for this.

  
  
Posted 4 months ago

I think your idea is related to issue 71 on GitHub . As far as I undestand it, it is not very straightforward to do it currently.

  
  
Posted 4 months ago
462 Views
4 Answers
4 months ago
4 months ago
Tags