Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I'Ve Multiple Tasks Setup In A Complex Pipeline. How Can I;

Hi, I've multiple tasks setup in a complex pipeline. How can I;
Define prior to running the pipeline, which tasks to be running on which remote queue using which images? Make certain tasks in the pipeline run in the same container session, instead of spawning new container sessions? (To improve efficiency)

  
  
Posted 2 years ago
Votes Newest

Answers 3


What type of pipeline steps are you running? From task, decorator or function?

We were trying with 'from task' at the moment. But the question apply to all methods.

If they're all running on the same container why not make them the same task and do things in parallel?

The tasks were created by different teams and their tasks content is rather independent and modular. Usage of them is usually optional. For example, task1 performs 'image whitening', task2 performs 'image resize'.

  
  
Posted 2 years ago

Hi SubstantialElk6 ,

Define prior to running the pipeline, which tasks to be running on which remote queue using which images?

What type of pipeline steps are you running? From task, decorator or function?

Make certain tasks in the pipeline run in the same container session, instead of spawning new container sessions? (To improve efficiency)

If they're all running on the same container why not make them the same task and do things in parallel?

  
  
Posted 2 years ago

SubstantialElk6 ,

We were trying with 'from task' at the moment. But the question apply to all methods.

You can specify this using add_function_step(..., execution_queue="<QUEUE>")

Make certain tasks in the pipeline run in the same container session, instead of spawning new container sessions? (To improve efficiency)

I'm not sure this is possible currently. This could a be nice feature request. Maybe open a github request?

  
  
Posted 2 years ago