Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, I Currently Have Some Data Processing Scripts For Example:

Hi all,

I currently have some data processing scripts for example:

  • script_1 -> dataset_0, dataset_1
  • dataset_1 -> script_2 -> dataset_2
  • dataset_2 -> script_3 -> dataset_3
  • dataset_0 + dataset_3 -> script_4 -> dataset_4
    As you can see, I have multiple datasets produced from multiple scripts, some of the datasets have multiple dataset as parents.

What is the recommended way to use clearml pipelines? The docs contain an example of how to transform tasks to pipeline steps, however it does not have any examples for datasets.

  
  
Posted one year ago
Votes Newest

Answers 3


Hi @<1523701168822292480:profile|ExuberantBat52>
What do you mean by:

  • dataset_1 -> script_2 -> dataset_2a dataset creates a script ?
  
  
Posted one year ago

Not exactly, the dataset gets called in the script using Dataset.get() and the second dataset is an output dataset using Dataset.create().. Which means that dataset_1 is a parent dataset of dataset_2.

  
  
Posted one year ago

The way ClearML thinks about it is the execution graph would be something like:
script_1 -> script_2 -> script_3 ->

Where each script would have in/out, so that you can trace the usage.

Trying to combine the two into a single "execution" graph might not represent the orchestration process.

That said visualizing them could be done.
I mean in theory there is no reason why we could add those "datasets" as other types of building blocks, for visualization purposes only

(Of course this would only make sense if you are creating pipelines)

wdyt?

  
  
Posted one year ago
1K Views
3 Answers
one year ago
one year ago
Tags