Unanswered
Hi There, Currently I Have A Clearml Pipeline That Takes In A Bunch Of Parameters For Various Tasks And Passes These Parameters Via Parameter_Override For Every Pipe.Add_Step(). However, I Have A Lot Of Parameters, And So My Pipeline Code Is A Little Unwi
would that mean that multiple pre_callback()s would have to be defined for every add_step, since every step would have different configs? Sorry if there's something I'm missing, I'm still not quite good at working with ClearML yet.
Yes, you could have multiple callbacks, or you could check the name of each step via node.name
and map the name of the node to its config.
One idea would be to have only 1 pipeline config file, that would look like:
step_1:
# step_1 config
step_2:
# step_2 config
then load this config with something like config = hocon.load("config_file_path")
and in the callback you would do:node.job.task.set_configuration_object(config[node.name])
134 Views
0
Answers
11 months ago
11 months ago