Seems that api has changed pretty much since a few versions back.

Correct, notice that your old pipelines Tasks use the older package and will still work.

There seems to be no need in 

controller_task

 anymore, right?

Correct, you can just call pipeline.start() 🙂

The pipeline creates the tasks, but never executes/enqueues them (they are all in 

Draft

 mode). No DAG graph appears in 

RESULTS/PLOTS

 tab.

Which version are we talking here ? v1.1.2? or the latest from GitHub ?

Sorry for the delay

Not reproduced, but caught another error when running pipeline_from_tasks.py
Traceback (most recent call last): File "pipeline_from_tasks.py", line 31, in <module> pipe.add_step(name='stage_data', base_task_project='examples', base_task_name='pipeline step 1 dataset artifact') File "/home/kirillfish/.local/lib/python3.6/site-packages/clearml/automation/controller.py", line 276, in add_step base_task_project, base_task_name)) ValueError: Could not find base_task_project=examples base_task_name=pipeline step 1 dataset artifact

OK, I managed to launch the example and it works

. In short, I was not able to do it with 

Task.clone

 and 

Task.create

 , the behavior differs from what is described in docs and docstrings (this is another story - I can submit an issue on github later)

The easiest is to use task_ task_overrides
Then pass:
task_overrides = dict('script': dict(diff='', branch='main'))

MelancholyElk85 assuming we are running with clearml 1.1.1 , let's debug the pipeline and instead of pipeline start/wait/stop :
Let's do:
pipeline.start_locally(run_pipeline_steps_locally=False)

but never executes/enqueues them (they are all in 

Draft

 mode).

All pipeline steps are not enqueued ?
Is the pipeline controller itself running?

pipeline controller itself is stuck at running mode forever all step tasks are created but never enqueued
I can share some code

creates all the step tasks in draft mode and then stucks

But I still cannot launch my own pipelines

You are right, I had [None] as parents in one of the tasks. Now this error is gone

pipeline launches on the server anyway (appears on the web UI)

AgitatedDove14 any ideas? 😀

👍

Hmm that is odd.
Can you verify with the latest from GitHub?
Is this reproducible with the pipeline example code?

Agent 1.1.0
Python client 1.1.1

it has the same effect as start/wait/stop, kinda weird

MelancholyElk85 I'm assuming you have the agent setup and everything in the example code works, is that correct ?
Where is it failing on your pipelines ?

What's the difference between the example pipeeline and this code ?
Could it be the "parents" argument ? what is it?

👍
Okay But we should definitely output an error on that

The pipeline is initialized like this
pipe = PipelineController(project=cfg['pipe']['project_name'], name='pipeline-{}'.format(name_postfix), version='1.0.0', add_pipeline_tags=True) pipe.set_default_execution_queue('my-queue')
Then for each step I have a base task which I want to clone

step_base_task = Task.get_task(project_name=cfg[name]['base_project'], task_name=cfg[name]['base_name'])
I want to modify a lot of params (docker image, docker params, task params, requirements, repository, branch...). From what I saw, the API of how this can be done has changed multiple times over the last few versions. In short, I was not able to do it with Task.clone and Task.create , the behavior differs from what is described in docs and docstrings (this is another story - I can submit an issue on github later)

I ended up exporting task configuration, modifying it in-place and then importing it to another task

` export_data = step_base_task.export_task()
export_data['name'] = '{} {}'.format(name, name_postfix)
export_data['project_name'] = cfg[name]['base_project']
export_data['script']['diff'] = ''
export_data['script']['branch'] = 'main'
export_data['script']['repository'] = cfg[name]['repo']
export_data['script']['requirements'] = {}
export_data['container']['image'] = 'registry.gitlab.com/my-image:my-tag'
export_data['container']['arguments'] = '-v /root/data:/data'

task = Task.import_task(export_data) Then I take this newly created task and add it as a pipeline step (I don't want to clone it one more time though, thus clone_base_task=False ` )

pipe.add_step( name=name, base_task_id=task.id, parents=parents, parameter_override=dict(flattener(cfg[name]['override'])), cache_executed_step=True, clone_base_task=False )
After adding all the steps, I simply run

pipe.start() pipe.wait() pipe.stop()
And we have what we have

I can share some code

Please do 🙂