Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Hi All, Does Anybody Happen To Have Any Recent Experience Spinning Up Service Workers (Via Clearml Agent) With A Limited Number Of Instances/Workers? We'D Like To Use This Functionality To Prevent Our Self-Hosted Server From Being Overloaded However We

Hi all,

Does anybody happen to have any recent experience spinning up service workers (via clearml agent) with a limited number of instances/workers?

We'd like to use this functionality to prevent our self-hosted server from being overloaded however we get the following warning when providing the maximum number of workers after the --services-mode flag as suggested in the docs (e.g. clearml-agent daemon --queue default --foreground --services-mode 5 ):

clearml_agent: Warning: Maximum number of service instance not supported, removing limit.

I've had a little dig into the code and this seems to be the issue seems to be the issue (fails to set runtime properties):

        # store in runtime configuration,
        if max_num_instances and not self.set_runtime_properties(key='max_num_instances', value=max_num_instances):
            warning('Maximum number of service instance not supported, removing limit.')
            max_num_instances = -1

Seems that the endpoint for workers.set_runtime_properties no longer exists:

APIError: code 404: Unable to find endpoint for name workers.set_runtime_properties and version 2.23

Is this functionality no longer supported?

Posted 7 months ago
Votes Newest

Answers 7

Hi @<1523701087100473344:profile|SuccessfulKoala55> , we're using the pre-built EC2 AMI I believe. We did an update on this a few weeks back so hopefully still up to date! Here's the version info from the app:

WebApp: 1.9.2-317 • Server: 1.9.2-317 • API: 2.23

I noticed that workers.set_runtime_properties isn't in the API reference either?

Posted 7 months ago

I am no expert but from your example it seems that you aren't running in docker mode. Or is it just an example? services-mode is supported on docker mode only

Posted 7 months ago

@<1534706830800850944:profile|ZealousCoyote89> are you using a locally hosted server? If so, what's the server's version?

Posted 7 months ago

Hi @<1534706830800850944:profile|ZealousCoyote89> , this is a feature supported only in the scale and enterprise versions of ClearML (requires server-side support)

Posted 7 months ago

Missed your message there @<1544853721739956224:profile|QuizzicalFox36> - I've been receiving the same warning with the docker flag provided too unfortunately.

@<1523701087100473344:profile|SuccessfulKoala55> , that's a shame but thank you for the clarification. I'm not sure how much on an option the scale/enterprise tiers are for us right now so it sounds like we'll need to re-think this a little bit. Just to double check - there's no other way of us limiting the number of workers on a daemon in services mode with a self-hosted server?

Posted 7 months ago

@<1534706830800850944:profile|ZealousCoyote89> that's correct. You can of course spin up several agents (not in services more, probably with the --cpu-only flag) to service the same queue, each capable of servicing a single task.

Posted 7 months ago

Okay, thanks for your help!

Posted 7 months ago
7 Answers
7 months ago
7 months ago