Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Is There Any Example Showing How To Work With Nested Pipelines? In My Case I Have Several Functions Decorated With

Is there any example showing how to work with nested pipelines? In my case I have several functions decorated with PipelineDecorator . In a pipeline I call some of the decorated functions. In the main pipeline I want to work with the secondary pipeline and other functions decorated with PipelineDecorator . Does ClearMl allow this? I have not been able to get it to work.

  
  
Posted 2 years ago
Votes Newest

Answers 10


I mean to use a function decorated with PipelineDecorator.pipeline inside another pipeline decorated in the same way.
In the traceback attached below you can see that I am trying to use a component named user_config_creation inside the create_user_configs sub-pipeline. I have imported user_config_creation inside create_user_configs but a KeyError is raised (however I assume the function has been imported correctly because no ImportError or ModuleNotFoundError occurred). Any clue on what might be going on? BTW, executing_pipeline is the name of the main pipeline.
Traceback (most recent call last): File "/user/project/new_pipeline.py", line 104, in <module> executing_pipeline() File "/user/anaconda3/envs/myenv/lib/python3.9/site-packages/clearml/automation/controller.py", line 2213, in internal_decorator func(**pipeline_kwargs) File "/user/project/new_pipeline.py", line 58, in executing_pipeline users_config_filenames = create_user_configs( File "/user/anaconda3/envs/myenv/lib/python3.9/site-packages/clearml/automation/controller.py", line 2213, in internal_decorator func(**pipeline_kwargs) File "/user/project/new_pipeline.py", line 29, in create_user_configs users_config_filenames[name] = user_config_creation( File "/user/anaconda3/envs/myenv/lib/python3.9/site-packages/clearml/automation/controller.py", line 2058, in wrapper _node = cls._singleton._nodes[_name] KeyError: 'user_config_creation'

  
  
Posted 2 years ago

In the main pipeline I want to work with the secondary pipeline and other functions decorated with 

PipelineDecorator

. Does ClearMl allow this? I have not been able to get it to work.

Usually when we think about pipelines or pipelines, the nested pipeline is just another Task you are running in the DAG (where the target queue is the services queue).
When you say nested pipelines with decorators, what exactly do you have in mind ?

  
  
Posted 2 years ago

I mean to use a function decorated with 

PipelineDecorator.pipeline

 inside another pipeline decorated in the same way.

Ohh... so would it make sense to add "helper_functions" so that a function will be available in the step's context ?
Or maybe we need a new to support "standalone" decorator?! Currently to actually "launch" the function step, you have to call it from the "pipeline" main logic function, but, at least in theory, one could do without the Pipeline itself... I really like this idea 🙂
Let me check a few things ...

  
  
Posted 2 years ago

Just to get the full picture, are we expecting to see the newly created step (aka eager execution) on the original pipeline (i.e. as part od the DAG visualization) ?

  
  
Posted 2 years ago

I think it was just pushed, including nested call you have to use the new argument for the decorator, helper_function
https://github.com/allegroai/clearml/blob/400c6ec103d9f2193694c54d7491bb1a74bbe8e8/clearml/automation/controller.py#L2392

  
  
Posted 2 years ago

YEY!

  
  
Posted 2 years ago

Exactly, at first I was trying to call a component from another component, but it didn't work. Then I thought it would be more natural to do this using a pipeline, but it didn't recognize the user_config_creation function despite I imported it as I would do under PipelineDecorator.component . I really like the idea of enabling an argument to specify the components you are going to use in the pipeline so they are in the step's context! I will be eagerly waiting for that feature :D

  
  
Posted 2 years ago

Mmm what would be the implications of not being part of the DAG? I mean, how could that step be launched if it is not part of the execution graph?

  
  
Posted 2 years ago

Hi AgitatedDove14 ,
Any updates on the new ClearML release that fixes the bugs we mentioned in this thread? :)

  
  
Posted 2 years ago

Beautiful. I have tested the new functionality with several use cases and it works just as I expected. Excellent work, as usual :D

  
  
Posted 2 years ago
564 Views
10 Answers
2 years ago
one year ago
Tags