Which kind of access specifically? I handle permissions with IAM roles
I wanted to access an Azure Storage Blob via an SAS token string, since this is how we built the codebase.
Hi TartSeal39 ,
Using this value as an argv (i.e. using argparse) will store it in the task's hyperparameters section and will make it accessible when executing remotely
thank you so much SuccessfulKoala55 ! 🥳 Accessing the variables with sys.argv didn’t work, but using argparse is working just fine!
Me again, SuccessfulKoala55 : would you also see an option for achieving this without having the token/string logged into clearml in plaintext? With argparse it’s stored as hyperparameter unfortunately.. Could I somehow prevent the logging of argparse by deleting the parser or such?
Well, if it won't be logged, the agent won't be able to use it when running remotely...
no, it’s a command-line argument now. Bc if it was in the code, it would be logged by github and I wanted to avoid having an access token logged in plaintext. That’s why argparse seemed to be nice to transfer it via commandline and not via code.
I tried parsing - storing as environment variable - deleting the parser object, but this wouldn’t work unfortunately
Well, I think the paid version should have some way of handling this in a centralized way
You can maybe encrypt it some way, if you're concerned with the keys showing up in the UI
hm, I see. Thank you! Do you see an option of transferring a simple, untracked file with execute_remotely() ? If this config file wouldn’t be tracked by github (so not part of the repo) then it would solve my problem
Can I ask what type of secrets are we talking about?
sure 🙂 an access token for an Azure storage blob. I could also just do more granular tokens with a smaller lifespan, but I was wondering if there is another solution
I was just wondering if it was something already used by the Agent/SDK in which case it can be placed directly in the clearml.conf file on the machine running the experiments remotely
yes, I also saw that, but unfortunately the clearml.config of my remote machine is stored in Azure as well, so we’d have the same access/permission/authentication problem again.
But thank you for your input!
If you're running the agent in docker mode, and assuming you're running in GCP or some other cloud solution, you can theoretically use the custom bash script configuration option for the agent to pull a file from some sort of secrets vault solution provided by the cloud provider and place on the machine for the duration of the task execution (or set the secret in an env var)
yes, that would be the solution which would make the most sense, thanks a lot! 🙂