Hi,
I’ve recently started experimenting with ClearML and the various features it offers. I’m primarily working on creating different pipelines, and I’ve encountered an issue I’d appreciate your help with.
I’ve noticed that, somewhat inconsistently, the inputs to a pipeline step, which are passed from previous steps, are occasionally received as None . The pipeline steps are defined with the retry_on_failure=3 parameter, so after several automatic retry attempts, the inputs are eventually valid (though sometimes they still remain None after the maximum retries). For example, in a pipeline where the first step creates a dataset that is passed to the next step, occasionally the next step starts running with the dataset being None (though this issue occurs with other input types as well, not just Dataset ).
Additionally, sometimes the input is not None , but accessing the data within it results in an error. For example:

  File "/tmp/tmpg0ykiwm2.py", line 22, in my_step
    if (clearml_dataset.name == 'debug_dataset'):
  File "/usr/local/lib/python3.8/dist-packages/clearml/datasets/dataset.py", line 345, in name
    return self._task.get_project_name().partition("/.datasets/")[-1]
AttributeError: 'NoneType' object has no attribute 'partition'

I’m using add_function_step for my pipeline definition.
Any help or insights would be greatly appreciated.
Thanks in advance!