Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Ran Into Something That I'D Describe And An Error But I Want To Verify This To Be The Case First. The Error Seems To Be Produced If I Call

I ran into something that I'd describe and an error but I want to verify this to be the case first. The error seems to be produced if I call output_model.update_weights() from the output model BEFORE calling logger.report_single_value() . What's strange, in both cases, the metric I'm attempting to log is successfully registered with the web UI bit the error messages only appear if I update weights before recording the metric.

The error I'm getting:

2023-04-20 14:22:13,670 - clearml.Task - INFO - Completed model upload to <url filtered>
2023-04-20 14:22:19,835 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:25,208 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:30,579 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:35,954 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:41,324 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:46,695 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:52,055 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:22:57,420 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:02,789 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:08,156 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:13,509 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:18,868 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:24,226 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:29,610 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:34,985 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:40,352 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:45,724 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:51,087 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:23:56,449 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:24:01,841 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Invalid task id <task ID>)>
2023-04-20 14:24:05.731 | INFO     | src.train_pipeline:train_process:253 - Finished with process.

Note: I removed the URL and TaskID from the logs.

The order of the code and snippets of the code related to ClearML:

task = Task.init(
        project_name=cfg.clearml.project_name,
        task_name="Training",
        task_type=Task.TaskTypes.training,
        reuse_last_task_id=False,
    )

...

output_model = OutputModel(
            task=task,
            comment="This is the model trained by the LTV model.",
            tags=task.get_tags(),
            framework="Lifetimes",
        )
        # upload the model to ClearML
        output_model.update_weights(
            weights_filename=str(Path(cfg.paths.get("model_save"), f"{model_name}.pkl"))
        )

...

clearml_logger.report_single_value(
                name="LTV Total",
                value=data.data["LTV"].sum(),
            )

...

task.close()

If I reverse the order of the calls between the output_model.update_weights() and clearml_logger.report_single_value() I don't get the error. It doesn't appear to actually prevent the metric from being logged, I can see it in the run on the web client either way.

  
  
Posted one year ago
Votes Newest

Answers 3


Hi @<1545216070686609408:profile|EnthusiasticCow4> , what version of clearml & server are you using?

  
  
Posted one year ago

Hi @<1545216070686609408:profile|EnthusiasticCow4> ! This is a known bug, we will likely fix it in the next version

  
  
Posted one year ago

@<1523701070390366208:profile|CostlyOstrich36> ClearML: 1.10.1, I'm not self-hosting the server so whatever the current version is. Unless you mean the operating system?

@<1523701435869433856:profile|SmugDolphin23> Good to know.

  
  
Posted one year ago