Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Traceback (Most Recent Call Last): File "/Root/.Clearml/Venvs-Builds/Task_Repository/Slot-Filler.Git/2_Engineer_Prompts.Py", Line 159, In <Module> Trainer.Train() File "/Root/Miniconda3/Envs/Ml-Training/Lib/Python3.9/Site-Packages/Transformers/Tra

Traceback (most recent call last): File "/root/.clearml/venvs-builds/task_repository/slot-filler.git/2_engineer_prompts.py", line 159, in <module> trainer.train() File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/transformers/trainer.py", line 1527, in train return inner_training_loop( File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/transformers/trainer.py", line 1704, in _inner_training_loop self.control = self.callback_handler.on_train_begin(args, self.state, self.control) File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/transformers/trainer_callback.py", line 353, in on_train_begin return self.call_event("on_train_begin", args, state, control) File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/transformers/trainer_callback.py", line 397, in call_event result = getattr(callback, event)( File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/transformers/integrations.py", line 1355, in on_train_begin self.setup(args, state, model, tokenizer, **kwargs) File "/root/.clearml/venvs-builds/task_repository/slot-filler.git/callback.py", line 30, in setup self._clearml_task.connect(args, "Args") File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/clearml/task.py", line 1480, in connect return method(mutable, name=name) File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/clearml/task.py", line 3449, in _connect_object a_dict = self._connect_dictionary(a_dict, name) File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/clearml/task.py", line 3413, in _connect_dictionary flat_dict = self._arguments.copy_to_dict(flat_dict, prefix=name) File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/clearml/backend_interface/task/args.py", line 508, in copy_to_dict self._task.set_parameter((prefix or '') + k, v) File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/clearml/backend_interface/task/task.py", line 1281, in set_parameter self._set_parameters( File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/clearml/backend_interface/task/task.py", line 1246, in _set_parameters description=create_description(), File "/root/miniconda3/envs/ml-training/lib/python3.9/site-packages/clearml/backend_interface/task/task.py", line 1237, in create_description created_description += "Values:\n" + ",\n".join( TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'I have this TypeError only when I execute my script remotely, i.e. by task.execute_remotely(queue_name="quad_gpu") . What might be the cause for create_description to ever be None?

  
  
Posted one year ago
Votes Newest

Answers 5


I have attached full log. This error happened during starting some standard transformers training loop.

  
  
Posted one year ago

The error is somehow connected to reinitializing task twice, I don't know what's the "true" way of using transformer's ClearMLCallback within clearml pipeline.

  
  
Posted one year ago

Can you please elaborate more on what is happening in your code while this occurs, Can you add the full log?

  
  
Posted one year ago

The custom callback I have used is:
` class MyClearMLCallback(ClearMLCallback):
def init(self, *args, **kwargs):
self._task_name = kwargs.pop("task_name", None)
self._project_name = kwargs.pop("project_name", None)
super().init(*args, **kwargs)

def setup(self, args, state, model, tokenizer, **kwargs):
    if self._clearml is None:
        return
    if state.is_world_process_zero:
        logger.info("Automatic ClearML logging enabled.")
        if self._clearml_task is None:
            self._clearml_task = self._clearml.Task.init(
                project_name=self._project_name,
                task_name=self._task_name,
                auto_connect_frameworks={"tensorboard": False, "pytorch": False},
                output_uri=True,
            )
            self._initialized = True
            logger.info("ClearML Task has been initialized.")

        self._clearml_task.connect(args, "Args")
        if hasattr(model, "config") and model.config is not None:
            self._clearml_task.connect(model.config, "Model Configuration") `
  
  
Posted one year ago

Something is changed by executing the program through agent, because I executed exactly the same code on exactly the same docker image and it doesn't produce this error.

  
  
Posted one year ago
820 Views
5 Answers
one year ago
one year ago
Tags
Similar posts