It seems that it's possible to provide some configuration parameters when creating the instance through the webui. Which seems to be a reasonable solution for secrets that you want to keep for a longer time.