Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Guys, Following Up On This

Hi guys, following up on this https://allegroai-trains.slack.com/archives/CTK20V944/p1599135173096200?thread_ts=1599125260.076600&cid=CTK20V944 : I have a pipeline TaskA -> TaskB -> TaskC -> TaskD.
Task A, B, C worked but D failed. Now I need to restart the pipeline from TaskA, but I want tasks A, B and C to be skipped because they already run. But since they produced artifacts, Task.init will create new tasks and not reuse them.
Would it be possible to extend Task.init with a force_reuse that would enforce reusing these tasks?

  
  
Posted 4 years ago
Votes Newest

Answers 11


Hi JitteryCoyote63
The new pipeline is almost ready for release (0.16.2),
It actually contains this exact scenario support.
Check out the example, and let me know if it fits what you are looking for:
https://github.com/allegroai/trains/blob/master/examples/pipeline/pipeline_controller.py

  
  
Posted 4 years ago

continue_last_task is almost what I want, the only problem with it is that it will start the task even if the task is completed

  
  
Posted 4 years ago

AgitatedDove14 This looks awesome! Unfortunately this would require a lot of changes in my current code, for that project I found a workaround 🙂 But I will surely use it for the next pipelines I will build!

  
  
Posted 4 years ago

Basically what I did is:
` if task_name is not None:
project_name = parent_task.get_project_name()
task = Task.get_task(project_name, task_name)
if task is not None:
return task

Otherwise here I create the Task `

  
  
Posted 4 years ago

I think you solved it 🙂
How do you imagine a slicker interface ?

  
  
Posted 4 years ago

the only problem with it is that it will start the task even if the task is completed

What is the criteria ?

  
  
Posted 4 years ago

it should return the task regardless if it is complete or not

  
  
Posted 4 years ago

JitteryCoyote63 Great to hear 🙂
BTW:

Would it be possible to extend 

Task.init

  with a 

force_reuse

 that would enforce reusing these tasks

You can pass continue_last_task=True I think it should be equivalent to what you suggest

  
  
Posted 4 years ago

Not sure about that, I think you guys solved it with your PipelineController implementation. I would need to test it before giving any feedback 🙂

  
  
Posted 4 years ago

And I do that each time I want to create a subtask. This way I am sure to retrieve the task if it already exists

  
  
Posted 4 years ago