Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Noticed After Upgrading To The Latest Clearml That App Credentials Now Disappear On Restart. Is This An Intentional Design Choice? I'M In A Bit Of A Chicken-And-Egg Situation: Trying To Generate Valid Keys For

I noticed after upgrading to the latest clearml that App Credentials now disappear on restart. Is this an intentional design choice?

I'm in a bit of a chicken-and-egg situation: trying to generate valid keys for clearml-agent-services requires me to run the UI, but then restarting the compose stack deletes the keys.

  
  
Posted 4 months ago
Votes Newest

Answers 31


Hi everyone, me and my team are on 1.16 and we're experiencing the same credentials disappearing behaviour reported by @<1689446563463565312:profile|SmallTurkey79> . Can confirm that the issue appeared the moment we switched to 1.16.0 .

@<1689446563463565312:profile|SmallTurkey79> Did you find a way around it?
@<1523701087100473344:profile|SuccessfulKoala55> Any updates on this?

  
  
Posted 4 months ago

Version should be released on Sunday

  
  
Posted 4 months ago

thank you!

by any chance do you have insights into github.com/allegroai/clearml-server/issues/248 ? dont know if its related to this at all or not, but it is an issue I experienced after upgrading .

  
  
Posted 4 months ago

Nope still dealing with it .

Oddly enough when i spin up a new instance on the new version, it doesnt seem to happen

  
  
Posted 4 months ago

Hi @<1689446563463565312:profile|SmallTurkey79> , checking and will answer asap

  
  
Posted 4 months ago

I understand. Thanks for the update!

  
  
Posted 4 months ago

@<1689446563463565312:profile|SmallTurkey79> to answer your question, this was a malfunction of an automatic process in the api server designed to remove redundant credentials in order to improve security

  
  
Posted 3 months ago

Hi @<1689446563463565312:profile|SmallTurkey79> , apologies for the delay, a patch version 1.16.1 ware released (images pushed yesterday), please try it out and let us know if it works for you

  
  
Posted 4 months ago

I did manage to figure this out with

docker compose stop agent-services
docker compose up --force-recreate --no-deps -d agent-services

and running an export for the newly generated key.

still though, noticing restarts cause App Credentials to be lost.

  
  
Posted 4 months ago

this is not about storage access tokens . its about the App Credentials .
those things you set as CLEARML_API_KEY and SECRET so that clients can talk to the api

  
  
Posted 4 months ago

Hi @<1689446563463565312:profile|SmallTurkey79>

App Credentials now disappear on restart.

You mean in the web UI?

  
  
Posted 4 months ago

hmm I assume the reason is the cookie / storage changed?

  
  
Posted 4 months ago

yup. once again, rebooted and lost my credentials.
image

  
  
Posted 4 months ago

Hi @<1689446563463565312:profile|SmallTurkey79> , how exactly are you passing these credentials and with which API keys?

  
  
Posted 4 months ago

when I do a docker compose down; docker compose up -d ... these disappear.
to be clear... this was not happening before I upgraded to the latest version. That is why I am asking about this.
image
image

  
  
Posted 4 months ago

@<1689446563463565312:profile|SmallTurkey79> can you try using CLEARML_AGENT_ACCESS_KEY and CLEARML_AGENT_SECRET_KEY with the new version (in the docker compose)?

  
  
Posted 4 months ago

Hi @<1689446563463565312:profile|SmallTurkey79> , we identified the problem and working on the fix

  
  
Posted 4 months ago

Hi @<1689446563463565312:profile|SmallTurkey79> , a patch version is on its way out, we will notify shortly

  
  
Posted 4 months ago

perfect. thank you. I verified that this was indeed reproducible on 1.16.0 with a fresh deployment.

  
  
Posted 4 months ago

hello @<1523701087100473344:profile|SuccessfulKoala55>
I appreciate your help. Thank you. Do you happen to have any updates? We had another restart and lost the creds again. So our deployment is in a brittle state on this latest upgrade, and I'm going back to 1.15.1 until I hear back.

  
  
Posted 4 months ago

so, I tried this on a fresh deployment, and for some reason that stack allows me to restart without losing App Credentials.
It's just the one that I performed an update on.

  
  
Posted 4 months ago

as a backup plan: is there a way to have an API key set up prior to running docker compose up?

Not sure I follow, the clearml API pair is persistent across upgrades, and the storage access token are unrelated (i.e. also persistent), what am I missing?

  
  
Posted 4 months ago

i am definitely not seeing it persist after upgrading. previously it wasn't a problem on other upgrades

  
  
Posted 4 months ago

thank you!
out of curiosity: how come the clearml-webserver upgrades weren't included in this release? was it just to patch the api part of the codebase?

  
  
Posted 4 months ago

for now I'm just avoiding restarts of the service, but I do want to get to the bottom of it using a fresh instance.

as a backup plan: is there a way to have an API key set up prior to running docker compose up? Like, I need at least one set of credentials that I can reliably have remote agents use, one that I know persists across restarts and upgrades.

  
  
Posted 4 months ago

yeah. thats how I've been generating credentials for agents as well as for my dev environment .

  
  
Posted 4 months ago

App Credentials now persist (I upgraded 1.15.1 -> 1.16.1 and the same keys exist!)
thanks!

  
  
Posted 4 months ago

everything i just said comes from the screenshotted webpage and is regarding the CLEARML_API_ACCESS_KEY and CLEARML_API_SECRET_KEY env vars.

when i restart clearml server, the keys started disappearing . this was not the case before upgrading

  
  
Posted 4 months ago

I can confirm that simply switching back to 1.15.1 results in persistent "App Credentials" across restarts.

Literally just did :%s/1.16.0/1.15.1/g , restarted the stack under the older version, created creds, and restarted again... and found them sitting there. So I know my volume mounts and all are good. It's something about the upgrade that caused this.

There's an issue on github that seems to be related, but the discussion under it seems to have digressed. Should I open a new issue?

  
  
Posted 4 months ago

it's really frustrating, as I'm trying to debug server behavior (so I'm restarting often), and keep needing to re-create these.

  
  
Posted 4 months ago
12K Views
31 Answers
4 months ago
3 months ago
Tags