Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, We Are Using Clearml On Gcp K8S Using The Clearml-Helm-Charts, We Noticed That After Restarting The Apiserver Pod, The User Credentials We Added For The Secret.Credentials.Apiserver And Secret.Credentials.Tests In The Values.Yaml Are Being Deleted Any

Hi,
We are using clearml on gcp k8s using the clearml-helm-charts,
We noticed that after restarting the apiserver pod, the user credentials we added for the secret.credentials.apiserver and secret.credentials.tests in the values.yaml are being deleted
Any idea what we are doing wrong?

  
  
Posted one year ago
Votes Newest

Answers 31


image

  
  
Posted one year ago

checking more

  
  
Posted one year ago

O k, I’d like to test it more with you; credentials exposed in chart values are system ones and it’s better to not change them; let’s forget about them for now. If you create a new accesskey/secretkey pair in ui, you should use these ones in your agents and they shuld not get overwritten in any way; can you confirm it works without touching credentials section?

  
  
Posted one year ago

Not sure I understand, you are saying I should not create user credentials and add them in values.yaml at secret.credentials.apiserver and secret.credentials.tests. ?

  
  
Posted one year ago

regardless of this I probably need to add some more detailed explanations on credentials configs

  
  
Posted one year ago

y ou can siply generate in UI the keys

  
  
Posted one year ago

Exactly, these are system accounts

  
  
Posted one year ago

I will check it and update you

  
  
Posted one year ago

ty

  
  
Posted one year ago

For now we used a workaround and forked the helm charts repo and we changed in the agents deployment.yaml, instead of taking the key and secret from the clearml-conf secret we take them from another secret we created so the server does not “know” about this new key and secret and does not reset them

  
  
Posted one year ago

Hey ApprehensiveSeahorse83 , I didn’t forget about you, it’s just a busy time for me; will answer during the day after a couple of more tests on my testing env.

  
  
Posted one year ago

but the system account key and secret can’t be the same for every installation, no? i need to generate specific one for my installation, no?

  
  
Posted one year ago

and then use them in agents if they are external

  
  
Posted one year ago

but Agents are on same k8s?

  
  
Posted one year ago

ty for your patience

  
  
Posted one year ago

In this case I suggest to give a try to k8s-glue that is there by default in latest chart version

  
  
Posted one year ago

we already using glue to manage our gpu pods. The agents we use for the pipelines are simple cpu agent.

  
  
Posted one year ago

I’m going to investigate this specific use case and will get back to you

  
  
Posted one year ago

Interesting use case, maybe we can create multiple k8s agents for different queues

  
  
Posted one year ago

can we use multiple k8s-glue - one for cpu and one for gpu pods?

  
  
Posted one year ago

since the gpu is expensive we want the glue to manage the pods

  
  
Posted one year ago

yes

  
  
Posted one year ago

still need time because I have two very busy days

  
  
Posted one year ago

not urgent after we used the workaround

  
  
Posted one year ago

It should work I think :)

  
  
Posted one year ago

but I will try to find something good for you

  
  
Posted one year ago

thanks for your help

  
  
Posted one year ago

for now we used fixed number of cpu agents but it will be better if it was dynamic with glue agent

  
  
Posted one year ago

if they are in kubernetes you can simply use k8s glue

  
  
Posted one year ago

Hi ApprehensiveSeahorse83 , today we released clearml-agent chart that just installs glue agent. My suggestion is to disable k8s glue and any other agent from the clearml chart and install more than one clearml-agent chart in different namespaces. In this way you will be able to have k8s glue for every queue (cpu and gpu).

  
  
Posted one year ago
693 Views
31 Answers
one year ago
13 days ago
Tags
gcp
Similar posts