Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Team, How Would You, Or What Is The Clearml Recommended Practice For Organising Sub-Steps Inside Each Pipeline Step? For Example I Have Several Sources Of Data To Retrieve, Each Of Which Is Potentially A Sub Step As I Wanted To Group Them Into One Big

Hi Team, how would you, or what is the ClearML recommended practice for organising sub-steps inside each pipeline step?

For example I have several sources of data to retrieve, each of which is potentially a sub step as I wanted to group them into one big pipeline step called "ingestion" ( I don't think make those sub step a pipeline step a good idea anyway).

What would be a good practice you recommend? Having class gathering them all will not record all the dependencies in each sub step correctly, right? (local imports are in each sub step's function code)

@PipelineDecorator.Component()
def ingestion_step():
    from somewhere.ingestion import Ingestion
    ingestion = Ingestion()
    ingestion.run()  # Runs all the sub step inside this class

class Ingestion:
  def __init__(self):
       self.state1 = 'Ah'
       self.a_data = None
       self.b_data = None
  
  def retrieve_a(self):
      # This import is not visible to ClearML?
      from sources.a import ASource
      self.a_data = ASource.retrieve()
  
  
Posted 20 days ago
Votes Newest

Answers 2


Hi @<1882599179692281856:profile|FriendlyBluewhale89> ,

I suggest using the pipeline stages for doing such - None

  
  
Posted 20 days ago

Thank you @<1744891825086271488:profile|RoundElephant20> ; Stage seems to be an UI management tool? I can't find anywhere in the SDK that we have an API for Stage; unless Stage is itself a Task ( which manages all the steps in its own realm and which is not just a wrapper of a group of tasks, it might be worth a try; otherwise I'm a bit reluctant to create too many tasks, unless absolutely must-have ( thinking about complex feature engineering which will just be too much for a single function).

  
  
Posted 19 days ago
158 Views
2 Answers
20 days ago
19 days ago
Tags