Verified @<1643060801088524288:profile|HarebrainedOstrich43> RC will be out soon for you to test, thank you again for catching it, not sure how internal tests missed it (btw the pipeline is created it's just not shown in the right place due to some internal typo)

@<1523701435869433856:profile|SmugDolphin23> , looks good, I can see the pipeline again. Thank you for a quick fix!

@<1523701205467926528:profile|AgitatedDove14> , I am not sure if it is a side effect of this fix, but with the new version of clearml, when I run pipeline locally it is no longer being reported to the server. Each step is reported separately in experiments, but no new run is visible in the "Pipelines".

Hi @<1643060801088524288:profile|HarebrainedOstrich43>
try this RC let me know if it works 🙂

pip install clearml==1.13.3rc1

@<1643060801088524288:profile|HarebrainedOstrich43> we released 1.14.1 as an official version

The other order (with custom decorator above pipeline fails - just for you info

)

This is on "purpose" the pipeline decorator has to be the top decorator.
Glad it works!

@<1523701205467926528:profile|AgitatedDove14> , great, thank you for reacting so fast :)

@<1523701205467926528:profile|AgitatedDove14> , the issue is present with 1.13.3rc1 for me as well. With 1.13.2 pipeline is correctly reported.

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! The rc is now out and installable via pip install clearml==1.14.1rc0

Dear @<1523701205467926528:profile|AgitatedDove14> and @<1523701435869433856:profile|SmugDolphin23> , i think i have noticed another interesting effect of decorators. When pipeline method has a custom wrapper (decorator) and I trigger the pipeline from clearml UI, then parameters that are provided via the UI are not actually passed to the pipeline method. At the same time custom decorators used on pipeline components and simple functions works exactly the same both when I trigger pipeline locally from the code, or via clearml UI. In my setup i trigger pipeline from clearml UI and it is executed on another machine that runs docker container with clearml agent.

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! Could you please share some code that could help us reproduced the issue? I tried cloning, changing parameters and running a decorated pipeline but the whole process worked as expected for me.

@<1523701205467926528:profile|AgitatedDove14> thank you very much for update! I am looking forward to the RC :)

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! Thank you for reporting. We will get back to you as soon as we have something

Hi @<1643060801088524288:profile|HarebrainedOstrich43>
You are absolutely correct we just fixed nested decorators in pipeline a week ago, let me check if he RC is already out with a fix.

Hi @<1523701205467926528:profile|AgitatedDove14> , i got a strange behavior also with this RC. For pipeline itself and simple functions it works well as before. But for pipeline components I still get an error. Below are some examples based on the code I have shared initially (I have slightly modified it).

With the old clearml version 1.12.2.

If I remove @my_decorator from my_pipeline_component_with_wrapper pipeline runs successfully and produces following output:

--- main started
ClearML Task: ...
--- my_pipeline started
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline ended
ClearML results page: ...
### my_pipeline_component_with_wrapper foo
--- main ended

When I add @my_decorator to my_pipeline_component_with_wrapper pipeline fails:

--- main started
--- internal_decorator started
ClearML Task: created ...
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
--- my_pipeline_component_with_wrapper started
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
Traceback (most recent call last):
  File "C:\Users\XXXXX~1\AppData\Local\Temp\tmptvtlgewe.py", line 11, in <module>
    @my_decorator
     ^^^^^^^^^^^^
NameError: name 'my_decorator' is not defined
Setting pipeline controller Task as failed (due to failed steps) !
Traceback (most recent call last): ...

It also seems to play a role if I put @my_decorator above or under @PipelineDecorator.

• The first example had @my_decorator below @PipelineDecorator for my_pipeline.
• The second example was for @my_decorator placed above @PipelineDecorator for both piepline and pipeline component.As you might see for the pipeline instead of "--- my_pipeline started" i got "--- internal_decorator started"

If I place @my_decorator under @PipelineDecorator for both pipeline and components - following output is produced and process freezes:

--- main started
ClearML Task: created ...
--- my_pipeline started
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "...\decorators.py", line 7, in wrapper
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\pipelines.py", line 23, in main
    result = my_pipeline()
             ^^^^^^^^^^^^^
  File "...\.venv\Lib\site-packages\clearml\automation\controller.py", line 4339, in internal_decorator
    pipeline_result = func(**pipeline_kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "....\decorators.py", line 7, in wrapper
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\pipelines.py", line 14, in my_pipeline
    c = my_pipeline_component_with_wrapper(b) # <----- it fails here
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "....\.venv\Lib\site-packages\clearml\automation\controller.py", line 3906, in wrapper
    kwargs[inspect_func.args[i]] = v
           ~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
PS ... ClearML results page: ....
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended

With the new version (1.13.3rc1), when I place @my_decorator under pipeline decorator I get exactly the same error as in the second example. For the case when custom decorators are placed above pipeline decorator following output is produced (and it still fails):

--- main started
--- internal_decorator started
ClearML Task: ...
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
--- my_pipeline_component_with_wrapper started
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
ClearML results page: ...
--- my_pipeline_component_with_wrapper started
ClearML pipeline page: ....
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
Traceback (most recent call last):
  File "C:\Users\XXXXX~1\AppData\Local\Temp\tmpag35j399.py", line 31, in <module>
    @my_decorator
     ^^^^^^^^^^^^
NameError: name 'my_decorator' is not defined

So it looks like behavior changed, but still results in an error. Also it looks like the order (and number) of function calls is a bit chaotic 😢

@<1523701205467926528:profile|AgitatedDove14> , thank you, I am looking forward for any updates. By the way, I am running pipeline with a poetry script, but I do not think it is relevant to the use of decorators.

New RC hopefully solves it @<1643060801088524288:profile|HarebrainedOstrich43> could you check if it works for you now?

pip install clearml==1.14.0rc0

Hmm this is odd in deed, let me verify (thanks! @<1643060801088524288:profile|HarebrainedOstrich43> )

Hmm that is odd. Let me take a look and ask the guys. Thank you for quickly testing the RC! I'm hoping a new RC with a fix will be there tomorrow, if we can quickly replicate

Hi @<1523701205467926528:profile|AgitatedDove14> , thanks for the update and for the fix. This RC works well on my test code when I place custom decorator under pipeline decorator, which is great. The other order (with custom decorator above pipeline fails - just for you info 🙂 ) I will soon update my production code and will let you know if I notice any side effects.