What's the difference between the example pipeeline and this code ?
Could it be the "parents" argument ? what is it?
it has the same effect as start/wait/stop, kinda weird
👍
Okay But we should definitely output an error on that
But I still cannot launch my own pipelines
Seems that api has changed pretty much since a few versions back.
Correct, notice that your old pipelines Tasks use the older package and will still work.
There seems to be no need in
controller_task
anymore, right?
Correct, you can just call pipeline.start()
🙂
The pipeline creates the tasks, but never executes/enqueues them (they are all in
Draft
mode). No DAG graph appears in
RESULTS/PLOTS
tab.
Which version are we talking here ? v1.1.2? or the latest from GitHub ?
creates all the step tasks in draft mode and then stucks
OK, I managed to launch the example and it works
pipeline controller itself is stuck at running mode forever all step tasks are created but never enqueued
I can share some code
Sorry for the delay
Not reproduced, but caught another error when running pipeline_from_tasks.py
Traceback (most recent call last): File "pipeline_from_tasks.py", line 31, in <module> pipe.add_step(name='stage_data', base_task_project='examples', base_task_name='pipeline step 1 dataset artifact') File "/home/kirillfish/.local/lib/python3.6/site-packages/clearml/automation/controller.py", line 276, in add_step base_task_project, base_task_name)) ValueError: Could not find base_task_project=examples base_task_name=pipeline step 1 dataset artifact
You are right, I had [None]
as parents in one of the tasks. Now this error is gone
. In short, I was not able to do it with
Task.clone
and
Task.create
, the behavior differs from what is described in docs and docstrings (this is another story - I can submit an issue on github later)
The easiest is to use task_ task_overrides
Then pass:task_overrides = dict('script': dict(diff='', branch='main'))
pipeline launches on the server anyway (appears on the web UI)
MelancholyElk85 assuming we are running with clearml 1.1.1 , let's debug the pipeline and instead of pipeline start/wait/stop :
Let's do:pipeline.start_locally(run_pipeline_steps_locally=False)
but never executes/enqueues them (they are all in
Draft
mode).
All pipeline steps are not enqueued ?
Is the pipeline controller itself running?
The pipeline is initialized like thispipe = PipelineController(project=cfg['pipe']['project_name'], name='pipeline-{}'.format(name_postfix), version='1.0.0', add_pipeline_tags=True) pipe.set_default_execution_queue('my-queue')
Then for each step I have a base task which I want to clone
step_base_task = Task.get_task(project_name=cfg[name]['base_project'], task_name=cfg[name]['base_name'])
I want to modify a lot of params (docker image, docker params, task params, requirements, repository, branch...). From what I saw, the API of how this can be done has changed multiple times over the last few versions. In short, I was not able to do it with Task.clone
and Task.create
, the behavior differs from what is described in docs and docstrings (this is another story - I can submit an issue on github later)
I ended up exporting task configuration, modifying it in-place and then importing it to another task
` export_data = step_base_task.export_task()
export_data['name'] = '{} {}'.format(name, name_postfix)
export_data['project_name'] = cfg[name]['base_project']
export_data['script']['diff'] = ''
export_data['script']['branch'] = 'main'
export_data['script']['repository'] = cfg[name]['repo']
export_data['script']['requirements'] = {}
export_data['container']['image'] = 'registry.gitlab.com/my-image:my-tag'
export_data['container']['arguments'] = '-v /root/data:/data'
task = Task.import_task(export_data) Then I take this newly created task and add it as a pipeline step (I don't want to clone it one more time though, thus
clone_base_task=False ` )
pipe.add_step( name=name, base_task_id=task.id, parents=parents, parameter_override=dict(flattener(cfg[name]['override'])), cache_executed_step=True, clone_base_task=False )
After adding all the steps, I simply run
pipe.start() pipe.wait() pipe.stop()
And we have what we have
MelancholyElk85 I'm assuming you have the agent setup and everything in the example code works, is that correct ?
Where is it failing on your pipelines ?
Hmm that is odd.
Can you verify with the latest from GitHub?
Is this reproducible with the pipeline example code?