Maybe this is part of the paid version, but would be cool if each user (in the web UI) could define their own secrets,

Very cool (and actually how it works), but at the end someone needs to pay for salaries 😉

The S3 bucket credentials are defined on the agent, as the bucket is also running locally on the same machine - but I would love for the code to download and apply the file automatically!

I have an idea here, why not use the "docker bash script" argument for that ?
It could be a script always running at the beginning of each execution, or a script each user can add on top?
https://github.com/allegroai/clearml-agent/blob/3c8e0ae5dbf819a8125d0a8b8e70739e1e8b37fe/docs/clearml.conf#L145
wdyt?