i am definitely not seeing it persist after upgrading. previously it wasn't a problem on other upgrades
@<1689446563463565312:profile|SmallTurkey79> to answer your question, this was a malfunction of an automatic process in the api server designed to remove redundant credentials in order to improve security
yup. once again, rebooted and lost my credentials.
perfect. thank you. I verified that this was indeed reproducible on 1.16.0 with a fresh deployment.
Hi @<1689446563463565312:profile|SmallTurkey79> , apologies for the delay, a patch version 1.16.1 ware released (images pushed yesterday), please try it out and let us know if it works for you
Nope still dealing with it .
Oddly enough when i spin up a new instance on the new version, it doesnt seem to happen
Hi @<1689446563463565312:profile|SmallTurkey79> , we identified the problem and working on the fix
thank you!
out of curiosity: how come the clearml-webserver upgrades weren't included in this release? was it just to patch the api part of the codebase?
for now I'm just avoiding restarts of the service, but I do want to get to the bottom of it using a fresh instance.
as a backup plan: is there a way to have an API key set up prior to running docker compose up? Like, I need at least one set of credentials that I can reliably have remote agents use, one that I know persists across restarts and upgrades.
as a backup plan: is there a way to have an API key set up prior to running docker compose up?
Not sure I follow, the clearml API pair is persistent across upgrades, and the storage access token are unrelated (i.e. also persistent), what am I missing?
I did manage to figure this out with
docker compose stop agent-services
docker compose up --force-recreate --no-deps -d agent-services
and running an export for the newly generated key.
still though, noticing restarts cause App Credentials to be lost.
Hi @<1689446563463565312:profile|SmallTurkey79>
App Credentials now disappear on restart.
You mean in the web UI?
@<1689446563463565312:profile|SmallTurkey79> can you try using CLEARML_AGENT_ACCESS_KEY and CLEARML_AGENT_SECRET_KEY with the new version (in the docker compose)?
this is not about storage access tokens . its about the App Credentials .
those things you set as CLEARML_API_KEY and SECRET so that clients can talk to the api
yeah. thats how I've been generating credentials for agents as well as for my dev environment .
Hi everyone, me and my team are on 1.16 and we're experiencing the same credentials disappearing behaviour reported by @<1689446563463565312:profile|SmallTurkey79> . Can confirm that the issue appeared the moment we switched to 1.16.0 .
@<1689446563463565312:profile|SmallTurkey79> Did you find a way around it?
@<1523701087100473344:profile|SuccessfulKoala55> Any updates on this?
hello @<1523701087100473344:profile|SuccessfulKoala55>
I appreciate your help. Thank you. Do you happen to have any updates? We had another restart and lost the creds again. So our deployment is in a brittle state on this latest upgrade, and I'm going back to 1.15.1 until I hear back.
Hi @<1689446563463565312:profile|SmallTurkey79> , checking and will answer asap
hmm I assume the reason is the cookie / storage changed?
App Credentials now persist (I upgraded 1.15.1 -> 1.16.1 and the same keys exist!)
thanks!
everything i just said comes from the screenshotted webpage and is regarding the CLEARML_API_ACCESS_KEY and CLEARML_API_SECRET_KEY env vars.
when i restart clearml server, the keys started disappearing . this was not the case before upgrading
when I do a docker compose down; docker compose up -d ... these disappear.
to be clear... this was not happening before I upgraded to the latest version. That is why I am asking about this.
Hi @<1689446563463565312:profile|SmallTurkey79> , a patch version is on its way out, we will notify shortly
so, I tried this on a fresh deployment, and for some reason that stack allows me to restart without losing App Credentials.
It's just the one that I performed an update on.
it's really frustrating, as I'm trying to debug server behavior (so I'm restarting often), and keep needing to re-create these.
if there's a process I'm not understanding please clarify...
but
(a) i start up the compose stack, log in via web browser as a user . this is on a remote server .
(b) i go to settings and generate a credential
(c) i use that credential to set up my local dev env, editing my clearml.conf
(d) i repeat (b) and use that credential to start up a remote workers to serve queues .
am i misunderstanding something? if there's another way to generate credentials I'm not familiar with it .
thank you!
by any chance do you have insights into github.com/allegroai/clearml-server/issues/248 ? dont know if its related to this at all or not, but it is an issue I experienced after upgrading .
Hi @<1689446563463565312:profile|SmallTurkey79> , how exactly are you passing these credentials and with which API keys?