Hi all,
I'm generating pipelines from tasks and there's a step that requires providing outputs from the parent task to the child, but I'd like to pass it as a single list of values. I was planning to use parameter_override to pass a list of values, each from a different parent. Is this possible? So far when I've tried passing a list of replacement values it doesn't appear to perform the variable substitution as expected. By that I mean if I were to pass 'Hydra/clearml.dataset_id': '${stage_etl.parameters.General/dataset_id}'
, the value of ${stage_etl.parameters.General/dataset_id}
will be substituted for the ID for the dataset. However, if I try and pass {'Hydra/s3_paths': ['${stage_inference_UKI_12.artifacts.LTV_Data.url}', '${stage_inference_UKI_24.artifacts.LTV_Data.url}']}
, a list with multiple substitutions, it appears to not perform the substitutions. Any thoughts? Is there a better approach?