Answering myself for future interested users (at least GrumpySeaurchin29 I think you were interested):
You can "hide" (explained below) secrets directly in the agent 😁 :
When you start the agent listening to a specific queue (i.e. the services worker), you can specify additional environment variables by prefixing them to the execution, i.e. FOO='bar' clearml-agent daemon ....
Modify the example AWS autoscaler script - after the driver = AWSDriver.from_config(conf)
, inject any environment variables with e.g.driver.extra_vm_bash_script += "\nexport FOO='{foo}'\n".format(foo=os.environ.get('FOO'))
The autoscaler will pick up those environment variables from the worker that's executing it.The "hide" above is because obviously a malicious user with access to the machine running the agent could ps ef
and get those secrets directly.