Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello Guys, I'M Trying To Setup A Selfhosted Clearml Server Using Kubernetes (With The "Kubernetes Glue, Map Clearml Jobs Directly To K8S Jobs" Method), So Far Everything Kind Of Works But I Have The Following Issues That I Could Use A Little Help With If

Hello guys, I'm trying to setup a selfhosted Clearml server using Kubernetes (with the "Kubernetes Glue, map ClearML jobs directly to K8s jobs" method), so far everything kind of works but I have the following issues that I could use a little help with if anyone has experienced them before:

  • When looking at the documentation on how to setup Clearml-Agent with Kubernetese I didnt see the part on the agent service mode, is it possible to setup a services queue and an agent in service mode when using Kubernetese and how would one go about doing that?
  • From time to time when trying to acces to my projects, I'll get the following messages "Fetch Experiments failed" and "Fetch tags failed" and none of my projects and or tasks will be visible, I also might get the error message "Error 0 : You can't write against a read only replica." did anyone encounter the same issue and if so do you have any pointers on where to look to solve it?
    Thank you :)
  
  
Posted one year ago
Votes Newest

Answers 4


Hi @<1549564752575860736:profile|HappyTiger58> ,

is it possible to setup a services queue and an agent in service mode when using Kubernetese and how would one go about doing that?

There's actually no need to do that - the k8s glue agent is suitable both as a normal daemon agent and a services agent, it all depends on the specific k8s task pod template you define in the agent chart

From time to time when trying to acces to my projects, I'll get the following messages "Fetch Experiments failed" and "Fetch tags failed" and none of my projects and or tasks will be visible, I also might get the error message "Error 0 : You can't write against a read only replica." did anyone encounter the same issue and if so do you have any pointers on where to look to solve it?

This sounds like some database stability issue - @<1523701827080556544:profile|JuicyFox94> do you have any idea?

  
  
Posted one year ago

@<1523701827080556544:profile|JuicyFox94> Thank you for the pointers I'll look into our MongoDB setup!

  
  
Posted one year ago

You can’t write on readonly replica is about MongoDB. I guess you are using a multiple replica setup. In this case the mongodb dependency chart have a lot of parameters to tweak the system and maybe also an arbiter is good for you. But this is a huge topic regarding mongodb specific k8s setups.

  
  
Posted one year ago

Nice so if I understand correctly I should have a first k8s glue agent listening on the default queue that can execute tasks and I can also have a second k8s glue agent that listen to the services queue that will handle my pipelines? Also just to make sure that I setup everything correctly in the chart, should I modify the value "agentk8sglue.image.repository" or the value "agentk8sglue.defaultContainerImage" and what image should I use? Thank you!

  
  
Posted one year ago
990 Views
4 Answers
one year ago
one year ago
Tags