Hi @<1523701435869433856:profile|SmugDolphin23> , would that mean that multiple pre_callback()s would have to be defined for every add_step, since every step would have different configs? Sorry if there's something I'm missing, I'm still not quite good at working with ClearML yet.

Well if you have:

ret_obj = None
for in in range(5):
    ret_obj = step_x(ret_obj)

SInce the orchestration automatically determine the order of execution using the logic of return objects the controller will execute them sequentially.

However, if your steps don't have dependencies like this:

for i in range(5):
    step_x(...)

It will try to execute them concurrently

@<1523702000586330112:profile|FierceHamster54> That's probably a good idea yeah, my question is would PipelineDecorator still be okay if I have multiple iterations of certain steps? For example if I call

for i in range(5):
step_x(...)

And how would consolidating all these step_xs work?

The {$step.id} is the most viable way to reference that step imo @<1633638724258500608:profile|BitingDeer35>

Hi @<1633638724258500608:profile|BitingDeer35> ! You could attach the configuration using set_configuration_object None in a pre_execute_callback . The argument is set here: None

Basically, you would have something like:

def pre_callback(pipeline, node, params):
    node.job.task.set_configuration_object(config)

pipe.add_step(..., pre_execute_callback=pre_callback)

Why note define your pipeline using PipelineDecorator instead, then you'll be able to call each of your pipeline components in a very pythonic way

I see I see... I'll keep the decorator way to do it in mind; for these step configs, would it make sense if they are in the form of, for example {$pipeline.parameter} and {$step_1.id}? Or if not what is the way to go about referencing other steps?

 would that mean that multiple pre_callback()s would have to be defined for every add_step, since every step would have different configs? Sorry if there's something I'm missing, I'm still not quite good at working with ClearML yet.

Yes, you could have multiple callbacks, or you could check the name of each step via node.name and map the name of the node to its config.

One idea would be to have only 1 pipeline config file, that would look like:

step_1:
  # step_1 config
step_2:
  # step_2 config

then load this config with something like config = hocon.load("config_file_path")
and in the callback you would do:
node.job.task.set_configuration_object(config[node.name])