I ran into something that I'd describe and an error but I want to verify this to be the case first. The error seems to be produced if I call output_model.update_weights() from the output model BEFORE calling logger.report_single_value() . What's strange, in both cases, the metric I'm attempting to log is successfully registered with the web UI bit the error messages only appear if I update weights before recording the metric.

The error I'm getting:

2023-04-20 14:22:13,670 - clearml.Task - INFO - Completed model upload to <url filtered>
2023-04-20 14:22:19,835 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:25,208 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:30,579 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:35,954 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:41,324 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:46,695 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:52,055 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:57,420 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:02,789 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:08,156 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:13,509 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:18,868 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:24,226 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:29,610 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:34,985 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:40,352 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:45,724 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:51,087 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:56,449 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:24:01,841 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:24:05.731 | INFO     | src.train_pipeline:train_process:253 - Finished with process.

Note: I removed the URL and TaskID from the logs.

The order of the code and snippets of the code related to ClearML:

task = Task.init(
        project_name=cfg.clearml.project_name,
        task_name="Training",
        task_type=Task.TaskTypes.training,
        reuse_last_task_id=False,
    )

...

output_model = OutputModel(
            task=task,
            comment="This is the model trained by the LTV model.",
            tags=task.get_tags(),
            framework="Lifetimes",
        )
        # upload the model to ClearML
        output_model.update_weights(
            weights_filename=str(Path(cfg.paths.get("model_save"), f"{model_name}.pkl"))
        )

...

clearml_logger.report_single_value(
                name="LTV Total",
                value=data.data["LTV"].sum(),
            )

...

task.close()

If I reverse the order of the calls between the output_model.update_weights() and clearml_logger.report_single_value() I don't get the error. It doesn't appear to actually prevent the metric from being logged, I can see it in the run on the web client either way.