Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
[Auto Scaler / Api Client Does Not See Tasks In Queue]

[Auto scaler / API client does not see tasks in queue]
We had used the AWS auto scaler (based on the aws_autoscaler.py script in the repo) and it worked great.
Now we have implemented a GCP auto scaler script after we have switched clouds (happy to share that btw, it is very lightweight.)
However, the clearml.automation.auto_scaler.AutoScaler object does not seem to see the tasks in the specified queue - any idea why that is?

Specifically, I’ve tested the code:

autoscaler.api_client.queues.get_by_id("QUEUE ID").entries # Returns []
autoscaler.api_client.queues.get_all() # Successfully returns all the queues, including the "a100-auto-spot" one

I had queued 2 experiments into that queue and when I started an agent by hand using clearml-agent daemon --queue a100-auto-spot --gpus all --detached and it starts working on the tasks right away (still loving the Agent functionality btw 😉 )

Any help is appreciated!

Posted 5 months ago
Votes Newest

Answers 2

Hi @<1523702496097210368:profile|ScantChimpanzee51> , this is very strange 😕 ... can you confirm the queue ID you're sending is identical to the one returned by get_all?

Posted 5 months ago

Hi @<1523701087100473344:profile|SuccessfulKoala55> , sorry there was a mistake on my end - clearml.conf pointed to the wrong URL 🙈

Posted 5 months ago
2 Answers
5 months ago
5 months ago
Similar posts