Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Probably My Question Will Be About Something Other Than Core Concepts Of Clearml, But Something Connected I Think.... I Want To Schedule Tasks On Gpu And I Want Many Agents To Use Given Gpu At One Time. I Also Want To Avoid Situation When Gpu Will Be Out

Probably my question will be about something other than core concepts of ClearML, but something connected I think....

I want to schedule tasks on GPU and I want many agents to use given GPU at one time. I also want to avoid situation when GPU will be out of memory. But I also would like to use GPU as fully as possible. Do you have any tips on how to do that?

Posted one year ago
Votes Newest

Answers 2

Hi RoundMosquito25
The main problem here is there is no way to know before running the Task how much memory it would need ... And without that parameter maximizing GPUs is quite challenging. wdyt?

Posted one year ago

Yes, it is a good reason 🙂
Do you maybe know a tool that measures that during execution (to avoid looking on nvidia-smi during all training)?

So, suppose, that a task T uses 27% of GPU, means, that we can spawn 3 agents on this GPU (suppose that we will give them only task T). Does it make sense?

Posted one year ago