Hi @<1523701168822292480:profile|ExuberantBat52>
What do you mean by:
- dataset_1 -> script_2 -> dataset_2a dataset creates a script ?
Not exactly, the dataset gets called in the script using Dataset.get() and the second dataset is an output dataset using Dataset.create().. Which means that dataset_1 is a parent dataset of dataset_2.
The way ClearML thinks about it is the execution graph would be something like:
script_1 -> script_2 -> script_3 ->
Where each script would have in/out, so that you can trace the usage.
Trying to combine the two into a single "execution" graph might not represent the orchestration process.
That said visualizing them could be done.
I mean in theory there is no reason why we could add those "datasets" as other types of building blocks, for visualization purposes only
(Of course this would only make sense if you are creating pipelines)
wdyt?