Hi!
Can someone show me an example of how PipelineController.create_draft works? I'm trying to create a template of a pipeline to run it later but I can't get it to work. This is my attempt:
Script for pipeline draft creation:` from clearml import PipelineController

Create and configure the pipeline controller.

pipe = PipelineController(
name="Iris-LogisticRegression Pipeline",
project="Mocks",
target_project="Mocks/Components",
version="0.0.1",
add_pipeline_tags=True,
abort_on_failure=False,
)

Attach some parameters to the pipeline.

pipe.add_parameter("mode", default="Train")

Populate the pipeline with ML steps.

pipe.add_step(
name="stage_collect_dataset",
base_task_project="Mocks",
base_task_name="Pipeline step 1: Collect/Fetch the data",
execution_queue="default",
continue_on_fail=False,
cache_executed_step=False,
time_limit=None,
)
pipe.add_step(
name="stage_preprocess_dataset",
parents=[
"stage_collect_dataset",
],
base_task_project="Mocks",
base_task_name="Pipeline step 2: Preprocess the data",
execution_queue="default",
continue_on_fail=False,
cache_executed_step=False,
time_limit=None,
parameter_override={
"General/dataset_url": "${stage_collect_dataset.artifacts.dataset.url}",
"General/test_size": 0.25,
},
)
pipe.add_step(
name="stage_train_model",
parents=[
"stage_preprocess_dataset",
],
base_task_project="Mocks",
base_task_name="Pipeline step 3: Train the model",
execution_queue="default",
continue_on_fail=False,
cache_executed_step=False,
time_limit=None,
parameter_override={
"General/dataset_task_id": "${stage_preprocess_dataset.id}",
},
)

pipe.create_draft()

print("Pipeline's draft successfully created!") Script to clone, enqueue and run the pipeline: import datetime

from clearml import Task

EXECUTION_QUEUE = "controllers"

Create the experiment/task and connect ClearML with the current process.

master_task = Task.init(
project_name="Mocks",
task_name="Pipeline runner",
task_type=Task.TaskTypes.application,
)

Get a reference to the pipeline task.

pipeline_task = Task.get_task(
project_name=master_task.get_project_name(),
task_name="Iris-LogisticRegression Pipeline",
)

Clone the pipeline task.

This creates a task with status Draft whose parameters can be modified.

run_date = datetime.datetime.now()
cloned_task = Task.clone(
source_task=pipeline_task, name=f"{pipeline_task.name} - {run_date:%Y%m%d:%H%M}"
)

Enqueue the configured pipeline task for execution.

Task.enqueue(cloned_task.id, queue_name=EXECUTION_QUEUE)

print(f"Pipeline {cloned_task.name!r} has been sent to queue {EXECUTION_QUEUE!r}") `The base tasks I use are the same as in this https://github.com/allegroai/clearml/tree/master/examples/pipeline .

The result is the pipeline task starts running, but again returns to the draft state, without actually executing any of the steps.

I should clarify this is not a problem with the pipeline itself, since when I run it with 'start' instead of 'create_draft', it works just fine. However, my intention is ONLY to create it to be executed later on.

I'm using clearml v1.1.6 and clearml-agent v1.1.2.

Has anyone ever encountered this problem? 😐