Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello Guys, I'M Trying To Setup A Selfhosted Clearml Server Using Kubernetes (With The "Kubernetes Glue, Map Clearml Jobs Directly To K8S Jobs" Method), So Far Everything Kind Of Works But I Have The Following Issues That I Could Use A Little Help With If

Hello guys, I'm trying to setup a selfhosted Clearml server using Kubernetes (with the "Kubernetes Glue, map ClearML jobs directly to K8s jobs" method), so far everything kind of works but I have the following issues that I could use a little help with if anyone has experienced them before:

  • When looking at the documentation on how to setup Clearml-Agent with Kubernetese I didnt see the part on the agent service mode, is it possible to setup a services queue and an agent in service mode when using Kubernetese and how would one go about doing that?
  • From time to time when trying to acces to my projects, I'll get the following messages "Fetch Experiments failed" and "Fetch tags failed" and none of my projects and or tasks will be visible, I also might get the error message "Error 0 : You can't write against a read only replica." did anyone encounter the same issue and if so do you have any pointers on where to look to solve it?
    Thank you :)
  
  
Posted 11 months ago
Votes Newest

Answers 4


Hi @<1549564752575860736:profile|HappyTiger58> ,

is it possible to setup a services queue and an agent in service mode when using Kubernetese and how would one go about doing that?

There's actually no need to do that - the k8s glue agent is suitable both as a normal daemon agent and a services agent, it all depends on the specific k8s task pod template you define in the agent chart

From time to time when trying to acces to my projects, I'll get the following messages "Fetch Experiments failed" and "Fetch tags failed" and none of my projects and or tasks will be visible, I also might get the error message "Error 0 : You can't write against a read only replica." did anyone encounter the same issue and if so do you have any pointers on where to look to solve it?

This sounds like some database stability issue - @<1523701827080556544:profile|JuicyFox94> do you have any idea?

  
  
Posted 11 months ago

Nice so if I understand correctly I should have a first k8s glue agent listening on the default queue that can execute tasks and I can also have a second k8s glue agent that listen to the services queue that will handle my pipelines? Also just to make sure that I setup everything correctly in the chart, should I modify the value "agentk8sglue.image.repository" or the value "agentk8sglue.defaultContainerImage" and what image should I use? Thank you!

  
  
Posted 11 months ago

@<1523701827080556544:profile|JuicyFox94> Thank you for the pointers I'll look into our MongoDB setup!

  
  
Posted 11 months ago

You can’t write on readonly replica is about MongoDB. I guess you are using a multiple replica setup. In this case the mongodb dependency chart have a lot of parameters to tweak the system and maybe also an arbiter is good for you. But this is a huge topic regarding mongodb specific k8s setups.

  
  
Posted 11 months ago
511 Views
4 Answers
11 months ago
11 months ago
Tags