Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello All, When I Create A Pipeline (Using Pipeline Decorator) And Call The Function Executing_Pipeline(). It Create A Pipeline In The "Pipeline" Menu, With A Nice Dag Graph.

Hello all,
When I create a pipeline (using pipeline decorator) and call the function executing_pipeline(). It create a pipeline in the "Pipeline" menu, with a nice DAG graph.
` @PipelineDecorator.pipeline(
name='aaa',
project='aaa',
version='0.1',
pipeline_execution_queue='default',
)
def executing_pipeline():

code to call the pipeline steps `But when I create a TaskScheduler to schedule creating pipeline

from clearml.automation import TaskScheduler scheduler = TaskScheduler() scheduler.add_task( name='recurring train model', schedule_function=executing_pipeline, queue='default', day=1, month=1, execute_immediately=True ) scheduler.start_remotely(queue='default') scheduler.start()It does not create a pipeline in the "Pipeline" menu. Instead it create a "Scheduler" task running in "DevOps" project. And this scheduler task create the tasks to run the pipeline steps. Because it does not create a pipeline, I can't see the nice DAG graph anymore.

Does anyone know how to use the TaskScheduler to create a normal pipeline task? Thanks

  
  
Posted 2 years ago
Votes Newest

Answers 2


Hi GrittyCormorant73 , can you please provide a standalone snippet with instructions to play with?

  
  
Posted 2 years ago

Hi CostlyOstrich36 basically, I'm using this code to create a pipeline https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py
After I run the code. I can go to the ClearML server and click on "Pipelines" menu and see a new pipeline running, with a DAG graph showing all the steps.

Now I want to schedule it to run once a month. So in the main function, instead of calling execute_pipeline() , I create a scheduler:
from clearml.automation import TaskScheduler scheduler = TaskScheduler() scheduler.add_task( name='recurring pipeline', schedule_function=executing_pipeline, queue='default', day=1, month=1, target_project='My Project', execute_immediately=True ) scheduler.start_remotely(queue='default') scheduler.start()This code create a task "Scheduler" running in project "DevOps". The scheduler will then run each steps of the pipeline itself.
So to me, it does not create the pipeline task under "Pipelines" menu, and no DAG graph. The pipeline is still run by the scheduler, so I still got the output of all the steps. I'm just wondering if I can change setting so it create the pipeline task under "Pipelines" menu, as if I manually run the execute_pipeline() function.

  
  
Posted 2 years ago