Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi. I Have A Question About Pipelines And Their Generated Dependency Graphs. I Took The Code Of The Clearml Pipeline From Decorator Example:

Hi. I have a question about pipelines and their generated dependency graphs.
I took the code of the clearml pipeline from decorator example:
https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py
and I nested the https://github.com/allegroai/clearml/blob/b24ed1937cf8a685f929aef5ac0625449d29cb69/examples/pipeline/pipeline_from_decorator.py#L72-L88 in a loop that repeats 3 times within the pipeline function (leaving the rest unchanged, see first image attached).
Running this, I see some unexpected dependencies in the pipeline visualisation (second image)

Why do we have the connections marked in red "?" (in the third image) ?

  
  
Posted 2 years ago
Votes Newest

Answers 9


I think this should be a valid use of pipelines. for example - at some step I choose to sweep across several values of some parameter and the rest of the steps are duplicated for each value of that parameter.
The additional edges in the graph suggest that these steps somehow contain dependencies that I do not wish them to have.

  
  
Posted 2 years ago

(I see the same thing in some evaluation code that I've written so I thought I'd reproduce it in the standard example)

  
  
Posted 2 years ago

PanickyMoth78 thank you for the mock code, I can verify it reproduces the issue. It seem that for some reason (bug) when the same function is called multiple times it "collects" parents, hence the odd graph,
BTW: if you want to see exactly what is passed to the step you can press on the step's full_details, and see the hyperparameter section.
I'll make sure we fix this bug in the next RC.

  
  
Posted 2 years ago

feature request: tell me what gets passed along each edge of the pipeline graph

Nice! please feel free to add to GH issue 🙂
Hmm yes that is odd, let me see if I can reproduce

  
  
Posted 2 years ago

Sure. It is a minor change from the code in the clearml examples for pipelines.
I just repeat the last two pipeline steps from that code in a loop (x3)
https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py

  
  
Posted 2 years ago

The additional edges in the graph suggest that these steps somehow contain dependencies that I do not wish them to have.

PanickyMoth78 I think I understand what you are saying, but it is hard to see if there is a "bug" here or a feature...
Can you post the full code of the pipline?

  
  
Posted 2 years ago

I imagine that these phantom dependencies will prevent parallelization. Is there a workaround?

yes, they might... workaround might be a bit ugly but copy pasting the functions and changing the name
BTW: I'll check when is the next RC scheduled for, maybe it will already contain a fix 🤞

  
  
Posted 2 years ago

I imagine that these phantom dependencies will prevent parallelization. Is there a workaround?

  
  
Posted 2 years ago

feature request: tell me what gets passed along each edge of the pipeline graph

  
  
Posted 2 years ago
1K Views
9 Answers
2 years ago
one year ago
Tags
Similar posts