Hi All, Does Anybody Happen To Have Any Recent Experience Spinning Up Service Workers (Via Clearml Agent) With A Limited Number Of Instances/Workers? We'D Like To Use This Functionality To Prevent Our Self-Hosted Server From Being Overloaded However We

Answered

Hi all,

Does anybody happen to have any recent experience spinning up service workers (via clearml agent) with a limited number of instances/workers?

We'd like to use this functionality to prevent our self-hosted server from being overloaded however we get the following warning when providing the maximum number of workers after the --services-mode flag as suggested in the docs (e.g. clearml-agent daemon --queue default --foreground --services-mode 5 ):

clearml_agent: Warning: Maximum number of service instance not supported, removing limit.

I've had a little dig into the code and this seems to be the issue seems to be the issue (fails to set runtime properties):

        # store in runtime configuration,
        if max_num_instances and not self.set_runtime_properties(key='max_num_instances', value=max_num_instances):
            warning('Maximum number of service instance not supported, removing limit.')
            max_num_instances = -1

Seems that the endpoint for workers.set_runtime_properties no longer exists:

APIError: code 404: Unable to find endpoint for name workers.set_runtime_properties and version 2.23

Is this functionality no longer supported?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ZealousCoyote89
				
					0
					 × 1

Votes Newest

Answers 7

Hi @<1523701087100473344:profile|SuccessfulKoala55> , we're using the pre-built EC2 AMI I believe. We did an update on this a few weeks back so hopefully still up to date! Here's the version info from the app:

WebApp: 1.9.2-317 • Server: 1.9.2-317 • API: 2.23

I noticed that workers.set_runtime_properties isn't in the API reference either?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ZealousCoyote89
				
					0
					 × 1

Missed your message there @<1544853721739956224:profile|QuizzicalFox36> - I've been receiving the same warning with the docker flag provided too unfortunately.

@<1523701087100473344:profile|SuccessfulKoala55> , that's a shame but thank you for the clarification. I'm not sure how much on an option the scale/enterprise tiers are for us right now so it sounds like we'll need to re-think this a little bit. Just to double check - there's no other way of us limiting the number of workers on a daemon in services mode with a self-hosted server?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ZealousCoyote89
				
					0
					 × 1

@<1534706830800850944:profile|ZealousCoyote89> are you using a locally hosted server? If so, what's the server's version?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

I am no expert but from your example it seems that you aren't running in docker mode. Or is it just an example? services-mode is supported on docker mode only

  				
Posted 
	one year ago

					More
				  		
  Report
		
					QuizzicalFox36
				
					0
					 × 1

Okay, thanks for your help!

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ZealousCoyote89
				
					0
					 × 1

@<1534706830800850944:profile|ZealousCoyote89> that's correct. You can of course spin up several agents (not in services more, probably with the --cpu-only flag) to service the same queue, each capable of servicing a single task.

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Hi @<1534706830800850944:profile|ZealousCoyote89> , this is a feature supported only in the scale and enterprise versions of ClearML (requires server-side support)

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Write your answer

1K Views

7 Answers

one year ago