Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone, I Have A Question About Using

Hi everyone, i have a question about using custom decorators with clearml pipeline. Here is my setup:

decorators.py

def my_decorator(function):
    @wraps(function)
    def wrapper(*args, **kwargs):
        print(f'{function.__name__} started')
        result = function(*args, **kwargs) 
        print(f'{function.__name__} ended')
        return result
    return wrapper

utils.py

from decorators import my_decorator
@my_decorator	
def my_func() -> str:
	return 'foo'

pipeline_components.py

from decorators import my_decorator

@PipelineDecorator.component(
    return_values=["result"], cache=False, task_type=TaskTypes.testing
)
def my_pipeline_component_no_wrapper() -> str:
	from utils import my_func
	return my_func()

@PipelineDecorator.component(
    return_values=["result"], cache=False, task_type=TaskTypes.testing
)
@my_decorator
def my_pipeline_component_with_wrapper() -> str:
	from decorators import my_decorator
	return 'bar'

pipelines.py

from decorators import my_decorator
from pipeline_components import my_pipeline_component

@PipelineDecorator.pipeline(name=TASK_NAME, project=PROJECT_NAME, version=VERSION)
@my_decorator
def my_pipeline() -> str:
	from utils import my_func
	from pipeline_components import my_pipeline_component
	
	my_func()
	my_pipeline_component_no_wrapper()
    my_pipeline_component_with_wrapper() # <----- it fails here
	
	return 'foo bar'

Code (and code in the decorator) correctly executes for

  • start of my_pipeline
  • my_func in the pipeline
  • my_pipeline_component_no_wrapper and my_func in it. However when it reaches my_pipeline_component_with_wrapper () - following error happens:
Launching step [my_pipeline_component_with_wrapper]
Traceback (most recent call last):
  File "C:\Users\XXXXXX~1\AppData\Local\Temp\tmpxmh_y8xb.py", line 11, in <module>
    @my_decorator
     ^^^^^^^^^^^^^^^
NameError: name 'my_decorator' is not defined

So it looks like custom decorator works

  • on the pipeline level
  • on simple functions called within either pipeline or pipeline componentBut it fails for pipeline components themselves...

Am I doing something wrong here? Is there a way to make decorators work for pipeline components (so that i do not have to wrap their content is separate function)?

Many thanks in advance!

PS. I have already tried importing decorator both on file and function levels.

  
  
Posted one month ago
Votes Newest

Answers 20


@<1523701205467926528:profile|AgitatedDove14> thank you very much for update! I am looking forward to the RC :)

  
  
Posted one month ago

Hi @<1643060801088524288:profile|HarebrainedOstrich43>
try this RC let me know if it works 🙂

pip install clearml==1.13.3rc1
  
  
Posted one month ago

Hi @<1523701205467926528:profile|AgitatedDove14> , i got a strange behavior also with this RC. For pipeline itself and simple functions it works well as before. But for pipeline components I still get an error. Below are some examples based on the code I have shared initially (I have slightly modified it).

With the old clearml version 1.12.2.

If I remove @my_decorator from my_pipeline_component_with_wrapper pipeline runs successfully and produces following output:

--- main started
ClearML Task: ...
--- my_pipeline started
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline ended
ClearML results page: ...
### my_pipeline_component_with_wrapper foo
--- main ended

When I add @my_decorator to my_pipeline_component_with_wrapper pipeline fails:

--- main started
--- internal_decorator started
ClearML Task: created ...
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
--- my_pipeline_component_with_wrapper started
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
Traceback (most recent call last):
  File "C:\Users\XXXXX~1\AppData\Local\Temp\tmptvtlgewe.py", line 11, in <module>
    @my_decorator
     ^^^^^^^^^^^^
NameError: name 'my_decorator' is not defined
Setting pipeline controller Task as failed (due to failed steps) !
Traceback (most recent call last): ...

It also seems to play a role if I put @my_decorator above or under @PipelineDecorator.

  • The first example had @my_decorator below @PipelineDecorator for my_pipeline.
  • The second example was for @my_decorator placed above @PipelineDecorator for both piepline and pipeline component.As you might see for the pipeline instead of "--- my_pipeline started" i got "--- internal_decorator started"

If I place @my_decorator under @PipelineDecorator for both pipeline and components - following output is produced and process freezes:

--- main started
ClearML Task: created ...
--- my_pipeline started
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "...\decorators.py", line 7, in wrapper
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\pipelines.py", line 23, in main
    result = my_pipeline()
             ^^^^^^^^^^^^^
  File "...\.venv\Lib\site-packages\clearml\automation\controller.py", line 4339, in internal_decorator
    pipeline_result = func(**pipeline_kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "....\decorators.py", line 7, in wrapper
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\pipelines.py", line 14, in my_pipeline
    c = my_pipeline_component_with_wrapper(b) # <----- it fails here
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "....\.venv\Lib\site-packages\clearml\automation\controller.py", line 3906, in wrapper
    kwargs[inspect_func.args[i]] = v
           ~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
PS ... ClearML results page: ....
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended

With the new version (1.13.3rc1), when I place @my_decorator under pipeline decorator I get exactly the same error as in the second example. For the case when custom decorators are placed above pipeline decorator following output is produced (and it still fails):

--- main started
--- internal_decorator started
ClearML Task: ...
### my_pipeline
--- my_func started
### my_func from my_pipeline
--- my_func ended
Launching step [my_pipeline_component_no_wrapper]
--- my_pipeline_component_with_wrapper started
ClearML results page: ...
### my_pipeline_component_no_wrapper foo
--- my_func started
### my_func from my_pipeline_component_no_wrapper
--- my_func ended
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
ClearML results page: ...
--- my_pipeline_component_with_wrapper started
ClearML pipeline page: ....
Launching step [my_pipeline_component_with_wrapper]
--- my_pipeline_component_with_wrapper ended
Traceback (most recent call last):
  File "C:\Users\XXXXX~1\AppData\Local\Temp\tmpag35j399.py", line 31, in <module>
    @my_decorator
     ^^^^^^^^^^^^
NameError: name 'my_decorator' is not defined

So it looks like behavior changed, but still results in an error. Also it looks like the order (and number) of function calls is a bit chaotic 😢

  
  
Posted one month ago

Hi @<1643060801088524288:profile|HarebrainedOstrich43>
You are absolutely correct we just fixed nested decorators in pipeline a week ago, let me check if he RC is already out with a fix.

  
  
Posted one month ago

@<1523701205467926528:profile|AgitatedDove14> , thank you, I am looking forward for any updates. By the way, I am running pipeline with a poetry script, but I do not think it is relevant to the use of decorators.

  
  
Posted one month ago

Hmm that is odd. Let me take a look and ask the guys. Thank you for quickly testing the RC! I'm hoping a new RC with a fix will be there tomorrow, if we can quickly replicate

  
  
Posted one month ago

Hi @<1523701205467926528:profile|AgitatedDove14> , thanks for the update and for the fix. This RC works well on my test code when I place custom decorator under pipeline decorator, which is great. The other order (with custom decorator above pipeline fails - just for you info 🙂 ) I will soon update my production code and will let you know if I notice any side effects.

  
  
Posted one month ago

@<1523701205467926528:profile|AgitatedDove14> , I am not sure if it is a side effect of this fix, but with the new version of clearml, when I run pipeline locally it is no longer being reported to the server. Each step is reported separately in experiments, but no new run is visible in the "Pipelines".

  
  
Posted one month ago

The other order (with custom decorator above pipeline fails - just for you info

)

This is on "purpose" the pipeline decorator has to be the top decorator.
Glad it works!

  
  
Posted one month ago

New RC hopefully solves it @<1643060801088524288:profile|HarebrainedOstrich43> could you check if it works for you now?

pip install clearml==1.14.0rc0
  
  
Posted one month ago

@<1523701205467926528:profile|AgitatedDove14> , the issue is present with 1.13.3rc1 for me as well. With 1.13.2 pipeline is correctly reported.

  
  
Posted one month ago

Verified @<1643060801088524288:profile|HarebrainedOstrich43> RC will be out soon for you to test, thank you again for catching it, not sure how internal tests missed it (btw the pipeline is created it's just not shown in the right place due to some internal typo)

  
  
Posted one month ago

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! The rc is now out and installable via pip install clearml==1.14.1rc0

  
  
Posted one month ago

Hmm this is odd in deed, let me verify (thanks! @<1643060801088524288:profile|HarebrainedOstrich43> )

  
  
Posted one month ago

@<1523701205467926528:profile|AgitatedDove14> , great, thank you for reacting so fast :)

  
  
Posted one month ago

@<1523701435869433856:profile|SmugDolphin23> , looks good, I can see the pipeline again. Thank you for a quick fix!

  
  
Posted one month ago

@<1643060801088524288:profile|HarebrainedOstrich43> we released 1.14.1 as an official version

  
  
Posted one month ago

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! Could you please share some code that could help us reproduced the issue? I tried cloning, changing parameters and running a decorated pipeline but the whole process worked as expected for me.

  
  
Posted one month ago

Dear @<1523701205467926528:profile|AgitatedDove14> and @<1523701435869433856:profile|SmugDolphin23> , i think i have noticed another interesting effect of decorators. When pipeline method has a custom wrapper (decorator) and I trigger the pipeline from clearml UI, then parameters that are provided via the UI are not actually passed to the pipeline method. At the same time custom decorators used on pipeline components and simple functions works exactly the same both when I trigger pipeline locally from the code, or via clearml UI. In my setup i trigger pipeline from clearml UI and it is executed on another machine that runs docker container with clearml agent.

  
  
Posted one month ago

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! Thank you for reporting. We will get back to you as soon as we have something

  
  
Posted one month ago