Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi,

Hi, SuccessfulKoala55 TimelyPenguin76 . Two K8s integration flavours section in  https://github.com/allegroai/clearml-agent  confuses me a lot. It introduces a downside(No real scheduling (k8s scheduler), no docker image verification (post-mortem only)) when Kubernetes map ClearML jobs directly to K8s jobs. I have two questions:
No real scheduling. Is it mean to use k8s scheduler rather than ClearML orchestration? I read the source codes of k8s.py(clear-agent/clear_agent/glue). But I can not find the meaning of no real scheduling, like specify the nodeName of Pod no docker image verification. Could you explain it in detail?Thanks in advance!

  
  
Posted 2 years ago
Votes Newest

Answers 9


Thanks for your reply 😀 !

  
  
Posted 2 years ago

Hi FloppyDeer99

What is the meaning of no real scheduling

I think the meaning is that from the moment a k8s job is created, the k8s is in charge of actually spinning the container. Since k8s has no real priority/order the scheduling order is not guaranteed form this point.

The idea of the cleaml-k8s -glue is that the glue will launch a job on the k8s cluster only if it is sure there are enough resources to actually spin the job now (as opposed to, sometime in the future), this means the priority and order are kept on the cleaml queue, as the glue will not pop a job unless it will be executed momentarily.

Make sense ?

  
  
Posted 2 years ago

K8s can schedule pod with different priorities.

I'm not sure I agree here, could you refer me to the docs on this ability in k8s ?

So maybe no real scheduling means there is no ClearML scheduling after applying pod to k8s.

That is correct 🙂

Does it will implement in the future?

Yes, this is enterprise feature, in the community you can specify --max-pods limit (which will cause it never to pull a job if it hits the max-pod limit)

  
  
Posted 2 years ago

Since k8s has no real priority/order the scheduling order is not guaranteed form this point.

K8s can schedule pod with different priorities. So maybe no real scheduling means there is no ClearML scheduling after applying pod to k8s.

only if it is sure there are enough resources to actually spin the job now

In 1.0.2 version, I do not find any logic about checking k8s resource in k8s.py. Does it will implement in the future?

  
  
Posted 2 years ago

Regarding #1, there is not scheduling once execution reaches the k8s cluster, but the scheduling is done before that, using the glue.
I think that in the paid tier there is support for dynamic GPU scheduling that allows you to specify allocation requirements on the task.

  
  
Posted 2 years ago

And no real scheduling in README exactly means that ClearML will not do scheduling for task which may provided in paid tier? In other words, Kubernetes will schedule the Pod for me. Is it right?

  
  
Posted 2 years ago

What is meaning of ‘the scheduling is done before that’? Does it mean the ClearML will schedule the task for me?

  
  
Posted 2 years ago

README introduces that no real scheduling when using Kubernetes. But I think no matter launch a Pod or Job that the Pod will be scheduled through k8s scheduler exception specifying the nodeName in Pod specification. This confuses me a lot. What is the meaning of no real scheduling I don’t understand no verification of docker image

  
  
Posted 2 years ago

Hi FloppyDeer99 ,
I'm not sure what you mean by "real" - using the k8s glue you can control the pod/job template and using that enforce node selectors etc. What do you mean by docker image verification?All in all, it all seems like stuff that can be easily added and/or modified

  
  
Posted 2 years ago
520 Views
9 Answers
2 years ago
one year ago
Tags