Hi Everyone, I Have A Question About Using

Answered

Hi everyone, i have a question about using custom decorators with clearml pipeline. Here is my setup:

decorators.py

def my_decorator(function):
    @wraps(function)
    def wrapper(*args, **kwargs):
        print(f'{function.__name__} started')
        result = function(*args, **kwargs) 
        print(f'{function.__name__} ended')
        return result
    return wrapper

utils.py

from decorators import my_decorator
@my_decorator	
def my_func() -> str:
	return 'foo'

pipeline_components.py

from decorators import my_decorator

@PipelineDecorator.component(
    return_values=["result"], cache=False, task_type=TaskTypes.testing
)
def my_pipeline_component_no_wrapper() -> str:
	from utils import my_func
	return my_func()

@PipelineDecorator.component(
    return_values=["result"], cache=False, task_type=TaskTypes.testing
)
@my_decorator
def my_pipeline_component_with_wrapper() -> str:
	from decorators import my_decorator
	return 'bar'

pipelines.py

from decorators import my_decorator
from pipeline_components import my_pipeline_component

@PipelineDecorator.pipeline(name=TASK_NAME, project=PROJECT_NAME, version=VERSION)
@my_decorator
def my_pipeline() -> str:
	from utils import my_func
	from pipeline_components import my_pipeline_component
	
	my_func()
	my_pipeline_component_no_wrapper()
    my_pipeline_component_with_wrapper() # <----- it fails here
	
	return 'foo bar'

Code (and code in the decorator) correctly executes for

start of my_pipeline
my_func in the pipeline
my_pipeline_component_no_wrapper and my_func in it. However when it reaches my_pipeline_component_with_wrapper () - following error happens:

Launching step [my_pipeline_component_with_wrapper]
Traceback (most recent call last):
  File "C:\Users\XXXXXX~1\AppData\Local\Temp\tmpxmh_y8xb.py", line 11, in <module>
    @my_decorator
     ^^^^^^^^^^^^^^^
NameError: name 'my_decorator' is not defined

So it looks like custom decorator works

on the pipeline level
on simple functions called within either pipeline or pipeline componentBut it fails for pipeline components themselves...

Am I doing something wrong here? Is there a way to make decorators work for pipeline components (so that i do not have to wrap their content is separate function)?

Many thanks in advance!

PS. I have already tried importing decorator both on file and function levels.

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					HarebrainedOstrich43
				
					0
					 × 1

Votes Newest

Answers 20

@<1523701205467926528:profile|AgitatedDove14> thank you very much for update! I am looking forward to the RC :)

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					HarebrainedOstrich43
				
					0
					 × 1

Hi @<1523701205467926528:profile|AgitatedDove14> , i got a strange behavior also with this RC. For pipeline itself and simple functions it works well as before. But for pipeline components I still get an error. Below are some examples based on the code I have shared initially (I have slightly modified it).

With the old clearml version 1.12.2.

If I remove @my_decorator from my_pipeline_component_with_wrapper pipeline runs successfully and produces following output:

--- main started
ClearML Task: ...
--- my_pipeline started
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline ended
ClearML results page: ...
### my_pipeline_component_with_wrapper foo
--- main ended

When I add @my_decorator to my_pipeline_component_with_wrapper pipeline fails:

--- main started
--- internal_decorator started
ClearML Task: created ...
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
--- my_pipeline_component_with_wrapper started
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
Traceback (most recent call last):
  File "C:\Users\XXXXX~1\AppData\Local\Temp\tmptvtlgewe.py", line 11, in <module>
    @my_decorator
     ^^^^^^^^^^^^
NameError: name 'my_decorator' is not defined
Setting pipeline controller Task as failed (due to failed steps) !
Traceback (most recent call last): ...

It also seems to play a role if I put @my_decorator above or under @PipelineDecorator.

The first example had @my_decorator below @PipelineDecorator for my_pipeline.
The second example was for @my_decorator placed above @PipelineDecorator for both piepline and pipeline component.As you might see for the pipeline instead of "--- my_pipeline started" i got "--- internal_decorator started"

If I place @my_decorator under @PipelineDecorator for both pipeline and components - following output is produced and process freezes:

--- main started
ClearML Task: created ...
--- my_pipeline started
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "...\decorators.py", line 7, in wrapper
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\pipelines.py", line 23, in main
    result = my_pipeline()
             ^^^^^^^^^^^^^
  File "...\.venv\Lib\site-packages\clearml\automation\controller.py", line 4339, in internal_decorator
    pipeline_result = func(**pipeline_kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "....\decorators.py", line 7, in wrapper
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\pipelines.py", line 14, in my_pipeline
    c = my_pipeline_component_with_wrapper(b) # <----- it fails here
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "....\.venv\Lib\site-packages\clearml\automation\controller.py", line 3906, in wrapper
    kwargs[inspect_func.args[i]] = v
           ~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
PS ... ClearML results page: ....
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended

With the new version (1.13.3rc1), when I place @my_decorator under pipeline decorator I get exactly the same error as in the second example. For the case when custom decorators are placed above pipeline decorator following output is produced (and it still fails):

--- main started
--- internal_decorator started
ClearML Task: ...
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
--- my_pipeline_component_with_wrapper started
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
ClearML results page: ...
--- my_pipeline_component_with_wrapper started
ClearML pipeline page: ....
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
Traceback (most recent call last):
  File "C:\Users\XXXXX~1\AppData\Local\Temp\tmpag35j399.py", line 31, in <module>
    @my_decorator
     ^^^^^^^^^^^^
NameError: name 'my_decorator' is not defined

So it looks like behavior changed, but still results in an error. Also it looks like the order (and number) of function calls is a bit chaotic 😢

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					HarebrainedOstrich43
				
					0
					 × 1

New RC hopefully solves it @<1643060801088524288:profile|HarebrainedOstrich43> could you check if it works for you now?

pip install clearml==1.14.0rc0

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! The rc is now out and installable via pip install clearml==1.14.1rc0

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

@<1643060801088524288:profile|HarebrainedOstrich43> we released 1.14.1 as an official version

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

@<1523701205467926528:profile|AgitatedDove14> , I am not sure if it is a side effect of this fix, but with the new version of clearml, when I run pipeline locally it is no longer being reported to the server. Each step is reported separately in experiments, but no new run is visible in the "Pipelines".

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					HarebrainedOstrich43
				
					0
					 × 1

Verified @<1643060801088524288:profile|HarebrainedOstrich43> RC will be out soon for you to test, thank you again for catching it, not sure how internal tests missed it (btw the pipeline is created it's just not shown in the right place due to some internal typo)

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

@<1523701205467926528:profile|AgitatedDove14> , the issue is present with 1.13.3rc1 for me as well. With 1.13.2 pipeline is correctly reported.

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					HarebrainedOstrich43
				
					0
					 × 1

@<1523701435869433856:profile|SmugDolphin23> , looks good, I can see the pipeline again. Thank you for a quick fix!

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					HarebrainedOstrich43
				
					0
					 × 1

@<1523701205467926528:profile|AgitatedDove14> , great, thank you for reacting so fast :)

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					HarebrainedOstrich43
				
					0
					 × 1

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! Could you please share some code that could help us reproduced the issue? I tried cloning, changing parameters and running a decorated pipeline but the whole process worked as expected for me.

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

Hi @<1523701205467926528:profile|AgitatedDove14> , thanks for the update and for the fix. This RC works well on my test code when I place custom decorator under pipeline decorator, which is great. The other order (with custom decorator above pipeline fails - just for you info 🙂 ) I will soon update my production code and will let you know if I notice any side effects.

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					HarebrainedOstrich43
				
					0
					 × 1

The other order (with custom decorator above pipeline fails - just for you info

)

This is on "purpose" the pipeline decorator has to be the top decorator.
Glad it works!

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Hi @<1643060801088524288:profile|HarebrainedOstrich43>
You are absolutely correct we just fixed nested decorators in pipeline a week ago, let me check if he RC is already out with a fix.

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Dear @<1523701205467926528:profile|AgitatedDove14> and @<1523701435869433856:profile|SmugDolphin23> , i think i have noticed another interesting effect of decorators. When pipeline method has a custom wrapper (decorator) and I trigger the pipeline from clearml UI, then parameters that are provided via the UI are not actually passed to the pipeline method. At the same time custom decorators used on pipeline components and simple functions works exactly the same both when I trigger pipeline locally from the code, or via clearml UI. In my setup i trigger pipeline from clearml UI and it is executed on another machine that runs docker container with clearml agent.

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					HarebrainedOstrich43
				
					0
					 × 1

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! Thank you for reporting. We will get back to you as soon as we have something

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

@<1523701205467926528:profile|AgitatedDove14> , thank you, I am looking forward for any updates. By the way, I am running pipeline with a poetry script, but I do not think it is relevant to the use of decorators.

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					HarebrainedOstrich43
				
					0
					 × 1

Hmm that is odd. Let me take a look and ask the guys. Thank you for quickly testing the RC! I'm hoping a new RC with a fix will be there tomorrow, if we can quickly replicate

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Hi @<1643060801088524288:profile|HarebrainedOstrich43>
try this RC let me know if it works 🙂

pip install clearml==1.13.3rc1

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Hmm this is odd in deed, let me verify (thanks! @<1643060801088524288:profile|HarebrainedOstrich43> )

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Write your answer

678 Views

20 Answers

10 months ago

9 months ago