. In short, I was not able to do it with
Task.clone
and
Task.create
, the behavior differs from what is described in docs and docstrings (this is another story - I can submit an issue on github later)
The easiest is to use task_ task_overrides
Then pass:task_overrides = dict('script': dict(diff='', branch='main'))
OK, I managed to launch the example and it works
it has the same effect as start/wait/stop, kinda weird
The pipeline is initialized like thispipe = PipelineController(project=cfg['pipe']['project_name'], name='pipeline-{}'.format(name_postfix), version='1.0.0', add_pipeline_tags=True) pipe.set_default_execution_queue('my-queue')
Then for each step I have a base task which I want to clone
step_base_task = Task.get_task(project_name=cfg[name]['base_project'], task_name=cfg[name]['base_name'])
I want to modify a lot of params (docker image, docker params, task params, requirements, repository, branch...). From what I saw, the API of how this can be done has changed multiple times over the last few versions. In short, I was not able to do it with Task.clone
and Task.create
, the behavior differs from what is described in docs and docstrings (this is another story - I can submit an issue on github later)
I ended up exporting task configuration, modifying it in-place and then importing it to another task
` export_data = step_base_task.export_task()
export_data['name'] = '{} {}'.format(name, name_postfix)
export_data['project_name'] = cfg[name]['base_project']
export_data['script']['diff'] = ''
export_data['script']['branch'] = 'main'
export_data['script']['repository'] = cfg[name]['repo']
export_data['script']['requirements'] = {}
export_data['container']['image'] = 'registry.gitlab.com/my-image:my-tag'
export_data['container']['arguments'] = '-v /root/data:/data'
task = Task.import_task(export_data) Then I take this newly created task and add it as a pipeline step (I don't want to clone it one more time though, thus
clone_base_task=False ` )
pipe.add_step( name=name, base_task_id=task.id, parents=parents, parameter_override=dict(flattener(cfg[name]['override'])), cache_executed_step=True, clone_base_task=False )
After adding all the steps, I simply run
pipe.start() pipe.wait() pipe.stop()
And we have what we have
pipeline controller itself is stuck at running mode forever all step tasks are created but never enqueued
I can share some code
Sorry for the delay
Not reproduced, but caught another error when running pipeline_from_tasks.py
Traceback (most recent call last): File "pipeline_from_tasks.py", line 31, in <module> pipe.add_step(name='stage_data', base_task_project='examples', base_task_name='pipeline step 1 dataset artifact') File "/home/kirillfish/.local/lib/python3.6/site-packages/clearml/automation/controller.py", line 276, in add_step base_task_project, base_task_name)) ValueError: Could not find base_task_project=examples base_task_name=pipeline step 1 dataset artifact
but never executes/enqueues them (they are all in
Draft
mode).
All pipeline steps are not enqueued ?
Is the pipeline controller itself running?
Hmm that is odd.
Can you verify with the latest from GitHub?
Is this reproducible with the pipeline example code?
pipeline launches on the server anyway (appears on the web UI)
MelancholyElk85 assuming we are running with clearml 1.1.1 , let's debug the pipeline and instead of pipeline start/wait/stop :
Let's do:pipeline.start_locally(run_pipeline_steps_locally=False)
What's the difference between the example pipeeline and this code ?
Could it be the "parents" argument ? what is it?
creates all the step tasks in draft mode and then stucks
You are right, I had [None]
as parents in one of the tasks. Now this error is gone
Seems that api has changed pretty much since a few versions back.
Correct, notice that your old pipelines Tasks use the older package and will still work.
There seems to be no need in
controller_task
anymore, right?
Correct, you can just call pipeline.start()
🙂
The pipeline creates the tasks, but never executes/enqueues them (they are all in
Draft
mode). No DAG graph appears in
RESULTS/PLOTS
tab.
Which version are we talking here ? v1.1.2? or the latest from GitHub ?
But I still cannot launch my own pipelines
👍
Okay But we should definitely output an error on that
MelancholyElk85 I'm assuming you have the agent setup and everything in the example code works, is that correct ?
Where is it failing on your pipelines ?