Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi There, Currently I Have A Clearml Pipeline That Takes In A Bunch Of Parameters For Various Tasks And Passes These Parameters Via Parameter_Override For Every Pipe.Add_Step(). However, I Have A Lot Of Parameters, And So My Pipeline Code Is A Little Unwi

Hi there, currently I have a clearml pipeline that takes in a bunch of parameters for various tasks and passes these parameters via parameter_override for every pipe.add_step(). However, I have a lot of parameters, and so my pipeline code is a little unwieldy with many many calls for pipe.add_parameter() and pipe.add_step(). Is there a more elegant way to manage this, perhaps with the parameters in a json/yml file and the adding of parameters/parameter overrides in a more lelegant way? Thank you in advance!

  
  
Posted 6 months ago
Votes Newest

Answers 8


 would that mean that multiple pre_callback()s would have to be defined for every add_step, since every step would have different configs? Sorry if there's something I'm missing, I'm still not quite good at working with ClearML yet.

Yes, you could have multiple callbacks, or you could check the name of each step via node.name and map the name of the node to its config.

One idea would be to have only 1 pipeline config file, that would look like:

step_1:
  # step_1 config
step_2:
  # step_2 config

then load this config with something like config = hocon.load("config_file_path")
and in the callback you would do:
node.job.task.set_configuration_object(config[node.name])

  
  
Posted 6 months ago

Hi @<1523701435869433856:profile|SmugDolphin23> , would that mean that multiple pre_callback()s would have to be defined for every add_step, since every step would have different configs? Sorry if there's something I'm missing, I'm still not quite good at working with ClearML yet.

  
  
Posted 6 months ago

@<1523702000586330112:profile|FierceHamster54> That's probably a good idea yeah, my question is would PipelineDecorator still be okay if I have multiple iterations of certain steps? For example if I call

for i in range(5):
step_x(...)

And how would consolidating all these step_xs work?

  
  
Posted 6 months ago

I see I see... I'll keep the decorator way to do it in mind; for these step configs, would it make sense if they are in the form of, for example {$pipeline.parameter} and {$step_1.id}? Or if not what is the way to go about referencing other steps?

  
  
Posted 6 months ago

Hi @<1633638724258500608:profile|BitingDeer35> ! You could attach the configuration using set_configuration_object None in a pre_execute_callback . The argument is set here: None

Basically, you would have something like:

def pre_callback(pipeline, node, params):
    node.job.task.set_configuration_object(config)

pipe.add_step(..., pre_execute_callback=pre_callback)
  
  
Posted 6 months ago

Why note define your pipeline using PipelineDecorator instead, then you'll be able to call each of your pipeline components in a very pythonic way

  
  
Posted 6 months ago

The {$step.id} is the most viable way to reference that step imo @<1633638724258500608:profile|BitingDeer35>

  
  
Posted 5 months ago

Well if you have:

ret_obj = None
for in in range(5):
    ret_obj = step_x(ret_obj)

SInce the orchestration automatically determine the order of execution using the logic of return objects the controller will execute them sequentially.

However, if your steps don't have dependencies like this:

for i in range(5):
    step_x(...)

It will try to execute them concurrently

  
  
Posted 6 months ago
291 Views
8 Answers
6 months ago
5 months ago
Tags
Similar posts