Dynamic pipelines in a notebook, so I don’t have to recreate a pipeline every time a step is changed 🤔
Oh nono, more like:
- Create a pipeline
- Add N steps to it
- Run the pipeline
- It fails/succeeds, the user does something with the output
- The user would like to add/modify some steps based on the results now (after closer inspection).I wonder if at (5), do I have to recreate the pipeline every time? 🤔
A step is changed while the pipeline is running?
See e None @<1523701087100473344:profile|SuccessfulKoala55>
Do you want to remove steps/add steps from the pipeline after it has ran basically? If that is the case, then it is theoretically possible, but we don't expose and methods that would allow you to do that...
What you would need to do is modify all the pipeline configuration entries you find in the CONFIGURATION section (see the screenshot), Not sure if that is worth the effort. I would simply create another version of the pipeline with the added/removed steps
Hi @<1523701083040387072:profile|UnevenDolphin73> ! Steps can be cached using the cache=True arg, so when a new step is added, the other steps will not be ran (only their outputs will be fetched). If a step is modified, then clearml will recognise that and not use the cached result. Of course, the whole pipeline will have to be reran, but the execution should be quick for the cached steps.
The pipeline can't be modified while it is running.
Does this help your case?
Yeah I was basically trying to avoid clutter in the Pipelines page. But see my other thread for the background, maybe you have some good input there? 🙏
Yeah I will probably end up archiving them for the time being (or deleting if possible?).
Otherwise (regarding the code question), I think it’s better if we continue the original thread, as it has a sample code snippet to illustrate what I’m trying to do.
btw, to avoid clutter you could also archive runs you don't need anymore
@<1523701083040387072:profile|UnevenDolphin73> are you composing the code you want to execute remotely by copy pasting it from various cells in one standalone cell?
Hey @<1523701435869433856:profile|SmugDolphin23> , thanks for the reply! I’m aware of the caching — that’s not the issue I’m trying to resolve 🙂
This is related to my other thread, so I’ll provide an example there -->
@<1523701083040387072:profile|UnevenDolphin73> at what point exactly? What's the scenario you're thinking of?