Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi. I Have A Question About Pipelines And Their Generated Dependency Graphs. I Took The Code Of The Clearml Pipeline From Decorator Example:

Hi. I have a question about pipelines and their generated dependency graphs.
I took the code of the clearml pipeline from decorator example:
https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py
and I nested the https://github.com/allegroai/clearml/blob/b24ed1937cf8a685f929aef5ac0625449d29cb69/examples/pipeline/pipeline_from_decorator.py#L72-L88 in a loop that repeats 3 times within the pipeline function (leaving the rest unchanged, see first image attached).
Running this, I see some unexpected dependencies in the pipeline visualisation (second image)

Why do we have the connections marked in red "?" (in the third image) ?

  
  
Posted one year ago
Votes Newest

Answers 9


I think this should be a valid use of pipelines. for example - at some step I choose to sweep across several values of some parameter and the rest of the steps are duplicated for each value of that parameter.
The additional edges in the graph suggest that these steps somehow contain dependencies that I do not wish them to have.

  
  
Posted one year ago

The additional edges in the graph suggest that these steps somehow contain dependencies that I do not wish them to have.

PanickyMoth78 I think I understand what you are saying, but it is hard to see if there is a "bug" here or a feature...
Can you post the full code of the pipline?

  
  
Posted one year ago

Sure. It is a minor change from the code in the clearml examples for pipelines.
I just repeat the last two pipeline steps from that code in a loop (x3)
https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py

  
  
Posted one year ago

(I see the same thing in some evaluation code that I've written so I thought I'd reproduce it in the standard example)

  
  
Posted one year ago

feature request: tell me what gets passed along each edge of the pipeline graph

  
  
Posted one year ago

PanickyMoth78 thank you for the mock code, I can verify it reproduces the issue. It seem that for some reason (bug) when the same function is called multiple times it "collects" parents, hence the odd graph,
BTW: if you want to see exactly what is passed to the step you can press on the step's full_details, and see the hyperparameter section.
I'll make sure we fix this bug in the next RC.

  
  
Posted one year ago

I imagine that these phantom dependencies will prevent parallelization. Is there a workaround?

yes, they might... workaround might be a bit ugly but copy pasting the functions and changing the name
BTW: I'll check when is the next RC scheduled for, maybe it will already contain a fix 🤞

  
  
Posted one year ago

feature request: tell me what gets passed along each edge of the pipeline graph

Nice! please feel free to add to GH issue 🙂
Hmm yes that is odd, let me see if I can reproduce

  
  
Posted one year ago

I imagine that these phantom dependencies will prevent parallelization. Is there a workaround?

  
  
Posted one year ago
615 Views
9 Answers
one year ago
one year ago
Tags
Similar posts