Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Hello I Have An Issue With The Queues. I Am Running Clearml Server + Agents In Kubernetes. Because Of That There Is A Default Internal Queue Preconfigured Called "K8S_Scheduler". I Have Defined Another Queue Called "Default" Where I Enqueue The Tasks. A


I have an issue with the queues. I am running ClearML server + agents in kubernetes. Because of that there is a default internal queue preconfigured called "k8s_scheduler".

I have defined another queue called "default" where I enqueue the tasks. As far as I understand the k8s glue pulls the task from the original queue (in this case "default") and places it in the "k8s_scheduler" queue from where the spawned worker pod will consume the task.

The problem is all my tasks have the "k8s_scheduler" queue set in the info tab (see screenshot). It makes it difficult to look back at a task and tell to which queue it was actually assigned.

This behavior also creates a problem when trying to launch pipelines from the UI. Here is what I do: I create a pipeline in the "draft" state with "set_default_execution_queue" set to the "default" queue. After that I go to the UI and click "NEW RUN". At this stage, looking at the "Advanced configuration" I can see the queue is indeed set to "default". I run the pipeline and it does get placed in the "default" queue as expected. Once it completes, I go and check the task that the pipeline created and the info shows "k8s_scheduler" as the queue, just like the other tasks.

Now I want to create a 2nd pipeline run from the UI with some different parameters. The behavior I see is that ClearML will use the information from the last successful pipeline task to populate the "NEW RUN" wizard and since the previous task info has the queue set to "k8s_scheduler" if I open "Advanced configuration" on this new pipeline run wizard the queue is indeed set to "k8s_scheduler" this time. The issue is basically that on every subsequent run of a pipeline you need to remember to check the "Advanced configuration" and set the queue to the appropriate one. So the "set_default_execution_queue" property does not seem to do anything after the pipeline had 1 successful run.

Posted one year ago
Votes Newest


If I right click on the initial pipeline Draft and hit "Run" from there, the new run wizard is populated with the default parameters value and uses "set_default_execution_queue" as the queue under "Advanced configuration".

Posted one year ago
1 Answer
one year ago
one year ago
Similar posts