Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone, I Had A Request To Use

Hi everyone, I had a request to use task_overrides during a pipeline step. The goal is to set the OUTPUT DESTINATION on the fly so:
pipe.add_step(name='create_dataset', base_task_project=project_name, base_task_name="Step 1 create dataset", task_overrides={"output.destination": " "}, execution_queue=args["worker_queue"])This is not working and we don't have any doc regarding a list of values we can override. How I can set it?

(bonus šŸ˜„ : the final goal should be to set output destination of every step in the pipeline with the value FROM the pipeline itself but I didn't find any property in "PipelineController" class about that šŸ˜ž )

  
  
Posted 3 years ago
Votes Newest

Answers 4


uh... yes, i was focusing on pipelinecontroller but it's a task property. Ty, it worked!

  
  
Posted 3 years ago

The PipelineController task? If so, you can get the task ( pipeine_task = Task.get_task(task_id=your pipeline task id) ) and after pipeine_task.get_output_destination() , can this do the trick?

  
  
Posted 3 years ago

output_dest works:)

About last point: I would like to retrieve the pipeline 'output_dest' and use it as a parameter in adding steps, is that possible?

  
  
Posted 3 years ago

Hi JuicyFox94 ,

Can you try with task_overrides={'output_dest':" s3://bucket/folder "} ?

This is not working and we donā€™t have any doc regarding a list of values we can override. How I can set it?

I agree, we should update the docs with some more information about it, and we will do šŸ™‚

(bonusĀ 

Ā : the final goal should be to set output destination of every step in the pipeline with the value FROM the pipeline itself but I didnā€™t find any property in ā€œPipelineControllerā€ class about thatĀ 

Ā )

Not sure I got that, you want to have the value for the steps from the Controller?

  
  
Posted 3 years ago