OK we found the source of the error @<1523701070390366208:profile|CostlyOstrich36> :

It's quite strange. Apparently, using the string "ON" as a key in the PipelineController's config, and connecting it via

pipeline.connect_configuration(config)

results in that key being processed as a boolean. so for example if we define

config = {
"ON": False,
"param1": True,
...
}

we end with that config defined inside the pipeline object like so

{
True: False,
"param1": True,
...
}

resulting in the error downstream

Hi @<1618780810947596288:profile|ExuberantLion50> , can you please a code snippet that reproduces this?

Here's the full traceback

  File "/Users/davidg/Projects/buzz/bumlo/buzz_ml/training_pipeline.py", line 85, in <module>
    pipe.start_locally(run_pipeline_steps_locally=True)
  File "/Users/davidg/opt/anaconda3/envs/bumlo/lib/python3.10/site-packages/clearml/automation/controller.py", line 985, in start_locally
    self._start(wait=True)
  File "/Users/davidg/opt/anaconda3/envs/bumlo/lib/python3.10/site-packages/clearml/automation/controller.py", line 1590, in _start
    self._prepare_pipeline(step_task_completed_callback, step_task_created_callback)
  File "/Users/davidg/opt/anaconda3/envs/bumlo/lib/python3.10/site-packages/clearml/automation/controller.py", line 1627, in _prepare_pipeline
    self.update_execution_plot()
  File "/Users/davidg/opt/anaconda3/envs/bumlo/lib/python3.10/site-packages/clearml/automation/controller.py", line 1321, in update_execution_plot
    self._update_execution_plot()
  File "/Users/davidg/opt/anaconda3/envs/bumlo/lib/python3.10/site-packages/clearml/automation/controller.py", line 2397, in _update_execution_plot
    self._force_task_configuration_update()
  File "/Users/davidg/opt/anaconda3/envs/bumlo/lib/python3.10/site-packages/clearml/automation/controller.py", line 2681, in _force_task_configuration_update
    description="pipeline state: {}".format(hash_dict(pipeline_dag)),
  File "/Users/davidg/opt/anaconda3/envs/bumlo/lib/python3.10/site-packages/clearml/storage/util.py", line 136, in hash_dict
    repr_string = json.dumps(a_dict, sort_keys=True)
  File "/Users/davidg/opt/anaconda3/envs/bumlo/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/Users/davidg/opt/anaconda3/envs/bumlo/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/davidg/opt/anaconda3/envs/bumlo/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
TypeError: '<' not supported between instances of 'str' and 'bool'