Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I’M

I’m working on an automated deployment of ClearML with IaC. I’ve got a script to start an EC2 instance that runs the docker compose file.

Separately, I’ve got an aws_autoscaler.yaml file that correctly configures aws_autoscaler.py with the right IAM permissions, startup commands, etc.

The missing piece I need to automate is: I need to add a single worker to the services queue when the EC2 instance starts up so that I can run the autoscaler task. I’ll run that worker on the same EC2 instance since the services queue should only handle light tasks.

But is there any way I can auto-generate the credentials needed for that first services worker/agent instance? Or is there a way I could get around creating them for that particular worker?

  
  
Posted one year ago
Votes Newest

Answers 10


The question I'm exploring remains: is it possible to acquire that initial set of ClearML API keys programmatically so that the manual steps of 1-4 above can be avoided for an initial deployment?

  
  
Posted one year ago

^^^ For my own notes: this is the web request made by the frontend to create a set of credentials
image

  
  
Posted one year ago

Oh my word, is this it? None

Can you set these to any strings and have them function as API keys?

  
  
Posted one year ago

I did a quick local experiment and observed that credentials created from the UI indeed become invalid if you delete the ClearML volumes.

  • starting docker-compose locally
  • creating a set of credentials from the UI
  • hardcodign those credentials into the docker-compose file
  • restarting
  • the agent-services container started up and successfully became a registered worker
  • I killed the docker-compose and deleted the volume folders
  • restarted the docker-compose (with the same hard-coded credentials)
    The resulting logs from the agent-services container were this:
Using environment access key CLEARML_API_ACCESS_KEY=NL0V1RKGB0TN57L3OZRK
Using environment secret key CLEARML_API_SECRET_KEY=********

clearml_agent: ERROR: Failed getting token (error 401 from 
): Unauthorized (invalid credentials) (failed to locate provided credentials)
  
  
Posted one year ago

This is already part of the docker-compose file,
https://github.com/allegroai/clearml-server/blob/master/docker/docker-compose.yml

  
  
Posted one year ago

One idea: is it possible to store usable credentials in advance and place them in a volume that the ClearML containers can access and then use?

  
  
Posted one year ago

I SOLVED IT, NO NEED TO READ FURTHER 😄

I'm a chump and didn't read the docs: None


Oh, I think I got overexcited and didn't look at this closely. So this ACCESS/SECRET key pair is on the agent-services container.

I can see that agent-services is simply a container running clearml-agent daemon --queue services --docker
and hey! that's what I was going to have to add to achieve getting a services worker!

I do have a question: Is there a way I can get the CLEARML_API_ACCESS_KEY and CLEARML_API_SECRET_KEY programmatically to give to this service? Or do I need to create these in the UI? If I create them in the UI and then destroy/re-create the EC2 instance the docker-compose.yml is running on, would that invalidate those keys?
image

  
  
Posted one year ago

That could work! Is that an option? Something that lets me spin up the ClearML and get a services worker to connect to it without manual steps.

  
  
Posted one year ago

yes 🙂

  
  
Posted one year ago

@<1541954607595393024:profile|BattyCrocodile47> you mean like environment variables?

  
  
Posted one year ago