Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey All, Quick Question About Pipeline Execution Queues. I Set The

Hey all, quick question about pipeline execution queues.
I set the execution_queue='queuename' on my individual pipeline components using the decorator, and can see in the UI under the pipeline configuration object that the queues are correctly configured.
However, all of the pipelines tasks are ran on the same queue. Could I be missing something?

  
  
Posted one year ago
Votes Newest

Answers 8


My bad, I worded my question wrong I see,

LOL no worries 🙂

Any chance you have some "debug" leftover in the Pipeline code:
https://github.com/allegroai/clearml/blob/7016138c849a4f8d0b4d296b319e0b23a1b7bd9e/examples/pipeline/pipeline_from_decorator.py#L113

Maybe we should show a warning when we it is being called, or ignore it when running via an agent ...

  
  
Posted one year ago

Hi ObedientDolphin41

However, all of the pipelines tasks are ran on the same queue. Could I be missing something?

The pipeline Task itself is running on a dedicated queue (meaning agent/s) usually because the pipeline logic is mostly idling, where as the components themselves are doing the actual compute.
Specifically you can control the pipeline logic queue with pipeline_execution_queue
https://github.com/allegroai/clearml/blob/7016138c849a4f8d0b4d296b319e0b23a1b7bd9e/clearml/automation/controller.py#L3593

Does that make sense ?

  
  
Posted one year ago

Not yet, working on running the autoscaler for now, and picking this up again later 🙂

  
  
Posted one year ago

Hi AgitatedDove14
My bad, I worded my question wrong I see, I meant the tasks of the pipeline’s components. (it shows that I’m a newbie 😅 )
This does make perfect sense though! The problem seems to just be that the components themselves are ran on the same queue as the pipeline logic, even though I configured it differently

  
  
Posted one year ago

This workflow however is the only way I have found to easily fix my previous ‘Module not found’ errors

Hmm okay make sense,
Did you try to set these ?
or even hack the sys.path with something like
import sys, os sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)+"/../")

  
  
Posted one year ago

Oh yup, that seems very possible since I run it with the run_locally() and then clone this task in the UI

  
  
Posted one year ago

This workflow however is the only way I have found to easily fix my previous ‘Module not found’ errors

  
  
Posted one year ago

This is ran by using the UI’s ‘Run’ button without the ‘Advanced configuration’

  
  
Posted one year ago
785 Views
8 Answers
one year ago
one year ago
Tags
Similar posts