Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Tasks Can Be Put In Draft State - If We Will Execute:

Tasks can be put in draft state - if we will execute:
task.execute_remotely()it will not start

Is there a way to do the same with pipeline (not run it immediately, but make it a draft and make it ready to be run in the future?

Posted one year ago
Votes Newest

Answers 9

You mean like sort of a stop period where you wait for additional input for pipeline to continue?

Posted one year ago

RoundMosquito25 it is true that the TaskScheduler requires a task_id , but that does not mean you have to run the pipeline every time 🙂

When setting up, you indeed need to run the pipeline once, to get it into the system. But from that point on, you should be able to just use the task_scheduler on the pipeline ID. The scheduler should automatically clone the pipeline and enqueue it. It will basically use the 1 existing pipeline as a "template" for subsequent runs.

Posted one year ago

The problem is that we have a a complex configuration of pipeline. Configuration changes quite frequently and we would not like to run the pipeline every time configuration changes, but we would like to have it scheduled in some defined periods.

Do you have an idea of some workaround / alternative solution for that problem?

Posted one year ago

That's what happens in the background when you click "new run". A pipeline is simply a task in the background. You can find the task using querying and you can clone it too! It is places in a "hidden" folder called .pipelines as a subfolder on your main project. Check out the settings, you can enable "show hidden folders"

Posted one year ago

Pipeline can also be in draft mode?

Posted one year ago

How do you run your pipeline currently?

Posted one year ago

It depends on how complex your configuration is, but if config elements are all that will change between versions (i.e. not the code itself) then you could consider using parameter overrides.

A ClearML Task can have a number of "hyperparameters" attached to it. But once that task is cloned and in draft mode, one can EDIT these parameters and change them. If then the task is queued, the new parameters will be injected into the code itself.

A pipeline is no different, it can have pipeline parameters that are accessible in the code. When cloning the pipeline using the TaskScheduler you can easily override these parameters 🙂

References: https://clear.ml/docs/latest/docs/pipelines/pipelines_sdk_tasks#pipeline-parameters

Posted one year ago

No. I would like to use TaskScheduler for pipelines. For now it seems to me, that I need to firstly run whole pipeline to get it's id.

I would like to define the pipeline but not run it before it is run by the scheduler

Posted one year ago
9 Answers
one year ago
one year ago