Answering myself for future interested users (at least GrumpySeaurchin29 I think you were interested):

You can "hide" (explained below) secrets directly in the agent 😁 :
When you start the agent listening to a specific queue (i.e. the services worker), you can specify additional environment variables by prefixing them to the execution, i.e. FOO='bar' clearml-agent daemon .... Modify the example AWS autoscaler script - after the driver = AWSDriver.from_config(conf) , inject any environment variables with e.g.driver.extra_vm_bash_script += "\nexport FOO='{foo}'\n".format(foo=os.environ.get('FOO'))The autoscaler will pick up those environment variables from the worker that's executing it.The "hide" above is because obviously a malicious user with access to the machine running the agent could ps ef and get those secrets directly.

the services queue (where the scaler runs) will be automatically exposed to new EC2 instance?

Yes, using this extra_clearml_conf parameter you can add configuration that will be passed to the clearml.conf of the instances it will spin.
Now an example to the values you want to add :
agent.extra_docker_arguments: ["-e", "ENV=value"]https://github.com/allegroai/clearml-agent/blob/a5a797ec5e5e3e90b115213c0411a516cab60e83/docs/clearml.conf#L149
wdyt?

That's fine as well - the code simply shows the name of the environment variable, not it's value, since that's taken directly from the agent listening to the services queue (and who's then running the scaler)

UnevenDolphin73 , if you're launching the Autoscaler through the apps, you can also add bash init script or additional configs - that's another way to inject env vars 🙂

Yes, that makes sense. Then you would need to use wither the AWS vault features, or the ClearML vault features ...

Or the above;)

Yes, using this extra_clearml_conf parameter you can add configuration

This is again exposing the environment variables on the WebUI for everyone to see.
The idea was to specify just the names of the environment variables, and that those would be exposed automatically to the EC2 instance, without specifying what values they should have (the value is taken from the agent running the scaler)

LOL yes 🙂
just make sure it won't be part of the uncommitted changes of the AWS autoscaler 😉

Nice!

CostlyOstrich36 I'm not sure what you mean by "through the apps", but any script AFAICS would expose the values of these environment variables; or what am I missing?

EmbarrassedSpider34 another way to go 🙂