But I still cannot launch my own pipelines

👍

Agent 1.1.0
Python client 1.1.1

Hmm that is odd.
Can you verify with the latest from GitHub?
Is this reproducible with the pipeline example code?

The pipeline is initialized like this
pipe = PipelineController(project=cfg['pipe']['project_name'], name='pipeline-{}'.format(name_postfix), version='1.0.0', add_pipeline_tags=True) pipe.set_default_execution_queue('my-queue')
Then for each step I have a base task which I want to clone

step_base_task = Task.get_task(project_name=cfg[name]['base_project'], task_name=cfg[name]['base_name'])
I want to modify a lot of params (docker image, docker params, task params, requirements, repository, branch...). From what I saw, the API of how this can be done has changed multiple times over the last few versions. In short, I was not able to do it with Task.clone and Task.create , the behavior differs from what is described in docs and docstrings (this is another story - I can submit an issue on github later)

I ended up exporting task configuration, modifying it in-place and then importing it to another task

` export_data = step_base_task.export_task()
export_data['name'] = '{} {}'.format(name, name_postfix)
export_data['project_name'] = cfg[name]['base_project']
export_data['script']['diff'] = ''
export_data['script']['branch'] = 'main'
export_data['script']['repository'] = cfg[name]['repo']
export_data['script']['requirements'] = {}
export_data['container']['image'] = 'registry.gitlab.com/my-image:my-tag'
export_data['container']['arguments'] = '-v /root/data:/data'

task = Task.import_task(export_data) Then I take this newly created task and add it as a pipeline step (I don't want to clone it one more time though, thus clone_base_task=False ` )

pipe.add_step( name=name, base_task_id=task.id, parents=parents, parameter_override=dict(flattener(cfg[name]['override'])), cache_executed_step=True, clone_base_task=False )
After adding all the steps, I simply run

pipe.start() pipe.wait() pipe.stop()
And we have what we have

. In short, I was not able to do it with 

Task.clone

 and 

Task.create

 , the behavior differs from what is described in docs and docstrings (this is another story - I can submit an issue on github later)

The easiest is to use task_ task_overrides
Then pass:
task_overrides = dict('script': dict(diff='', branch='main'))

pipeline controller itself is stuck at running mode forever all step tasks are created but never enqueued
I can share some code

I can share some code

Please do 🙂

AgitatedDove14 any ideas? 😀

You are right, I had [None] as parents in one of the tasks. Now this error is gone

pipeline launches on the server anyway (appears on the web UI)

creates all the step tasks in draft mode and then stucks

👍
Okay But we should definitely output an error on that

it has the same effect as start/wait/stop, kinda weird

Sorry for the delay

Not reproduced, but caught another error when running pipeline_from_tasks.py
Traceback (most recent call last): File "pipeline_from_tasks.py", line 31, in <module> pipe.add_step(name='stage_data', base_task_project='examples', base_task_name='pipeline step 1 dataset artifact') File "/home/kirillfish/.local/lib/python3.6/site-packages/clearml/automation/controller.py", line 276, in add_step base_task_project, base_task_name)) ValueError: Could not find base_task_project=examples base_task_name=pipeline step 1 dataset artifact

Seems that api has changed pretty much since a few versions back.

Correct, notice that your old pipelines Tasks use the older package and will still work.

There seems to be no need in 

controller_task

 anymore, right?

Correct, you can just call pipeline.start() 🙂

The pipeline creates the tasks, but never executes/enqueues them (they are all in 

Draft

 mode). No DAG graph appears in 

RESULTS/PLOTS

 tab.

Which version are we talking here ? v1.1.2? or the latest from GitHub ?

MelancholyElk85 I'm assuming you have the agent setup and everything in the example code works, is that correct ?
Where is it failing on your pipelines ?

but never executes/enqueues them (they are all in 

Draft

 mode).

All pipeline steps are not enqueued ?
Is the pipeline controller itself running?

MelancholyElk85 assuming we are running with clearml 1.1.1 , let's debug the pipeline and instead of pipeline start/wait/stop :
Let's do:
pipeline.start_locally(run_pipeline_steps_locally=False)

What's the difference between the example pipeeline and this code ?
Could it be the "parents" argument ? what is it?

OK, I managed to launch the example and it works