Unanswered
Clearml Pipelines Can Be Build From Tasks, Functions, And Decorated Functions, According To The Examples In
@<1523701083040387072:profile|UnevenDolphin73> : A big point for me is to reuse/cache those artifacts/datasets/models that need to be passed between the steps, but have been produced by colleagues' executions at some earlier point. So for example, let the pipeline be A(a) -> B(b) -> C(c), where A,B,C are steps and their code, excluding configurations/parameters, and a,b,c are the configurations/parameters. Then I might have the situation, that my colleague ran the pipeline A(a) -> B(b) -> C(c).
- Scenario 1: I run A(a) -> B(b') -> C(c) and I want that A(a) is not rerun, but its result reused/cached and only B(b') -> C(c') is run.
- Scenario 2: I run A(a) -> B'(b) -> C(c) and I want that A(a) is not rerun, but its result reused/cached and only B'(b) -> C(c) is run.
- Scenario 3: I run A(a) -> B'(b) -> C(c) and I want that nothing is rerun. Here, I only want that changes to the configuration, but not the code is considered.Which of the pipelines can be used for which Scenario?
(Yes, they are not academic hypothetical, I have those cases in real life.)
163 Views
0
Answers
one year ago
one year ago