Hi @<1523701205467926528:profile|AgitatedDove14> , thanks for the update and for the fix. This RC works well on my test code when I place custom decorator under pipeline decorator, which is great. The other order (with custom decorator above pipeline fails - just for you info 🙂 ) I will soon update my production code and will let you know if I notice any side effects.
Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! Could you please share some code that could help us reproduced the issue? I tried cloning, changing parameters and running a decorated pipeline but the whole process worked as expected for me.
Verified @<1643060801088524288:profile|HarebrainedOstrich43> RC will be out soon for you to test, thank you again for catching it, not sure how internal tests missed it (btw the pipeline is created it's just not shown in the right place due to some internal typo)
Hi @<1523701205467926528:profile|AgitatedDove14> , i got a strange behavior also with this RC. For pipeline itself and simple functions it works well as before. But for pipeline components I still get an error. Below are some examples based on the code I have shared initially (I have slightly modified it).
With the old clearml version 1.12.2.
If I remove @my_decorator from my_pipeline_component_with_wrapper pipeline runs successfully and produces following output:
--- main started
ClearML Task: ...
--- my_pipeline started
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline ended
ClearML results page: ...
### my_pipeline_component_with_wrapper foo
--- main ended
When I add @my_decorator to my_pipeline_component_with_wrapper pipeline fails:
--- main started
--- internal_decorator started
ClearML Task: created ...
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
--- my_pipeline_component_with_wrapper started
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
Traceback (most recent call last):
File "C:\Users\XXXXX~1\AppData\Local\Temp\tmptvtlgewe.py", line 11, in <module>
@my_decorator
^^^^^^^^^^^^
NameError: name 'my_decorator' is not defined
Setting pipeline controller Task as failed (due to failed steps) !
Traceback (most recent call last): ...
It also seems to play a role if I put @my_decorator above or under @PipelineDecorator.
- The first example had @my_decorator below @PipelineDecorator for my_pipeline.
- The second example was for @my_decorator placed above @PipelineDecorator for both piepline and pipeline component.As you might see for the pipeline instead of "--- my_pipeline started" i got "--- internal_decorator started"
If I place @my_decorator under @PipelineDecorator for both pipeline and components - following output is produced and process freezes:
--- main started
ClearML Task: created ...
--- my_pipeline started
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "...\decorators.py", line 7, in wrapper
result = function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "...\pipelines.py", line 23, in main
result = my_pipeline()
^^^^^^^^^^^^^
File "...\.venv\Lib\site-packages\clearml\automation\controller.py", line 4339, in internal_decorator
pipeline_result = func(**pipeline_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "....\decorators.py", line 7, in wrapper
result = function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "...\pipelines.py", line 14, in my_pipeline
c = my_pipeline_component_with_wrapper(b) # <----- it fails here
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "....\.venv\Lib\site-packages\clearml\automation\controller.py", line 3906, in wrapper
kwargs[inspect_func.args[i]] = v
~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
PS ... ClearML results page: ....
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
With the new version (1.13.3rc1), when I place @my_decorator under pipeline decorator I get exactly the same error as in the second example. For the case when custom decorators are placed above pipeline decorator following output is produced (and it still fails):
--- main started
--- internal_decorator started
ClearML Task: ...
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
--- my_pipeline_component_with_wrapper started
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
ClearML results page: ...
--- my_pipeline_component_with_wrapper started
ClearML pipeline page: ....
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
Traceback (most recent call last):
File "C:\Users\XXXXX~1\AppData\Local\Temp\tmpag35j399.py", line 31, in <module>
@my_decorator
^^^^^^^^^^^^
NameError: name 'my_decorator' is not defined
So it looks like behavior changed, but still results in an error. Also it looks like the order (and number) of function calls is a bit chaotic 😢
Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! The rc is now out and installable via pip install clearml==1.14.1rc0
@<1523701205467926528:profile|AgitatedDove14> thank you very much for update! I am looking forward to the RC :)
The other order (with custom decorator above pipeline fails - just for you info
)
This is on "purpose" the pipeline decorator has to be the top decorator.
Glad it works!
@<1523701435869433856:profile|SmugDolphin23> , looks good, I can see the pipeline again. Thank you for a quick fix!
@<1523701205467926528:profile|AgitatedDove14> , thank you, I am looking forward for any updates. By the way, I am running pipeline with a poetry script, but I do not think it is relevant to the use of decorators.
Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! Thank you for reporting. We will get back to you as soon as we have something
@<1643060801088524288:profile|HarebrainedOstrich43> we released 1.14.1 as an official version
Hi @<1643060801088524288:profile|HarebrainedOstrich43>
try this RC let me know if it works 🙂
pip install clearml==1.13.3rc1
@<1523701205467926528:profile|AgitatedDove14> , I am not sure if it is a side effect of this fix, but with the new version of clearml, when I run pipeline locally it is no longer being reported to the server. Each step is reported separately in experiments, but no new run is visible in the "Pipelines".
Hi @<1643060801088524288:profile|HarebrainedOstrich43>
You are absolutely correct we just fixed nested decorators in pipeline a week ago, let me check if he RC is already out with a fix.
Hmm that is odd. Let me take a look and ask the guys. Thank you for quickly testing the RC! I'm hoping a new RC with a fix will be there tomorrow, if we can quickly replicate
@<1523701205467926528:profile|AgitatedDove14> , the issue is present with 1.13.3rc1 for me as well. With 1.13.2 pipeline is correctly reported.
Dear @<1523701205467926528:profile|AgitatedDove14> and @<1523701435869433856:profile|SmugDolphin23> , i think i have noticed another interesting effect of decorators. When pipeline method has a custom wrapper (decorator) and I trigger the pipeline from clearml UI, then parameters that are provided via the UI are not actually passed to the pipeline method. At the same time custom decorators used on pipeline components and simple functions works exactly the same both when I trigger pipeline locally from the code, or via clearml UI. In my setup i trigger pipeline from clearml UI and it is executed on another machine that runs docker container with clearml agent.
New RC hopefully solves it @<1643060801088524288:profile|HarebrainedOstrich43> could you check if it works for you now?
pip install clearml==1.14.0rc0
@<1523701205467926528:profile|AgitatedDove14> , great, thank you for reacting so fast :)
Hmm this is odd in deed, let me verify (thanks! @<1643060801088524288:profile|HarebrainedOstrich43> )