I've noticed a change from ClearML 1.13.1 to 1.13.2 relating to the Hydra integration. I'm not yet convinced it's a bug but it could be but it also might just require a different approach because of the Hydra changes in the update. After upgrading to 1.13.2 it broke my pipelines, specifically the parameter_override option when overriding Hydra config groups. Hydra supports hierarchal configs and config groups. So instead of changing a setting in a config file, I could change the whole config file by updating the config group. I've been using this feature in my pipeline for a few versions now. For reference, I'm using hydra-core==1.3.2 .

Example Pipeline Code:

@hydra.main(version_base="1.3", config_path="../configs", config_name="pipeline.yaml")
def clearml_pipeline(cfg: DictConfig):
    pipe = PipelineController(
        name="Quarterly LTV Model Training and Inference",
        project=cfg.clearml.project_name,
        version="0.2.0",
        add_pipeline_tags=False,
    )
    # pipeline parameters
    pipe.add_parameter(
        name="countries",
        default=[
            "Global",
            # "Netherlands",
            # "Belgium",
            # "France",
            # "UKI",
            # "Spain",
            # "CEE",
            # "DACH",
            # "Nordics",
        ],
        description="A list of countries to build models for",
    )

    pipe.add_parameter(
        name="inference_period",
        default=[
            12,
            # 24,
            # 36,
            # 60,
        ],
        description="A list of periods (in months) to run inference for",
    )

    # Sets which queue on ClearML to pull agents for processing the pipeline steps
    pipe.set_default_execution_queue(cfg.get("default_execution_queue"))

    first_stage_name = "stage_etl"

    # the first stage is the ETL step to pass the data to the child steps
    pipe.add_step(
        name=first_stage_name,
        base_task_project=cfg.clearml.project_name,
        base_task_name="ETL Process",
    )

    # we need to iterate over the countries
    # second stage is training
    for country in pipe.get_parameters()["countries"]:
        if country is None:
            second_stage_name = "stage_training_global"
            third_stage_name = "stage_inference_global"
        else:
            second_stage_name = f"stage_training_{country}"
            third_stage_name_base = f"stage_inference_{country}"
        pipe.add_step(
            name=second_stage_name,
            base_task_project=cfg.clearml.project_name,
            parents=[first_stage_name],
            base_task_name="Training",
            parameter_override={
                # get the country from loop over the pipeline parameters
                "Hydra/countries": country,
                # Pass the data from the ETL step to the train step
                "Hydra/datagenerator": "clearml",
                "Hydra/clearml.dataset_id": f"${{{first_stage_name}.parameters.General/dataset_id}}",
            },
        )
...

When run with 1.13.1 the pipeline is happy as a clam. When run with 1.13.2 I get the following error with hydra when the second stage spins up:

Environment setup completed successfully
Starting Task Execution:
force-add of config groups is not supported: '++datagenerator=clearml'
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
2023-12-13 16:59:09
Process failed, exit code 1

This is because of "Hydra/datagenerator": "clearml", line in the pipeline_override 's in the second stage pipeline. I'm guessing something changed under the hood on how ClearML managed Hydra parameter overrides.