yes, I also saw that, but unfortunately the clearml.config of my remote machine is stored in Azure as well, so we’d have the same access/permission/authentication problem again.
But thank you for your input!
sure 🙂 an access token for an Azure storage blob. I could also just do more granular tokens with a smaller lifespan, but I was wondering if there is another solution
Can I ask what type of secrets are we talking about?
If you're running the agent in docker mode, and assuming you're running in GCP or some other cloud solution, you can theoretically use the custom bash script configuration option for the agent to pull a file from some sort of secrets vault solution provided by the cloud provider and place on the machine for the duration of the task execution (or set the secret in an env var)
Hi TartSeal39 ,
Using this value as an argv (i.e. using argparse) will store it in the task's hyperparameters section and will make it accessible when executing remotely
thank you so much SuccessfulKoala55 ! 🥳 Accessing the variables with sys.argv didn’t work, but using argparse is working just fine!
I wanted to access an Azure Storage Blob via an SAS token string, since this is how we built the codebase.
Which kind of access specifically? I handle permissions with IAM roles
no, it’s a command-line argument now. Bc if it was in the code, it would be logged by github and I wanted to avoid having an access token logged in plaintext. That’s why argparse seemed to be nice to transfer it via commandline and not via code.
I tried parsing - storing as environment variable - deleting the parser object, but this wouldn’t work unfortunately
Me again, SuccessfulKoala55 : would you also see an option for achieving this without having the token/string logged into clearml in plaintext? With argparse it’s stored as hyperparameter unfortunately.. Could I somehow prevent the logging of argparse by deleting the parser or such?
Well, if it won't be logged, the agent won't be able to use it when running remotely...
hm, I see. Thank you! Do you see an option of transferring a simple, untracked file with execute_remotely() ? If this config file wouldn’t be tracked by github (so not part of the repo) then it would solve my problem
You can maybe encrypt it some way, if you're concerned with the keys showing up in the UI
Well, I think the paid version should have some way of handling this in a centralized way
I was just wondering if it was something already used by the Agent/SDK in which case it can be placed directly in the clearml.conf file on the machine running the experiments remotely
yes, that would be the solution which would make the most sense, thanks a lot! 🙂