Indeed, running pipelines that were started with pipe.start_locally can not be cloned and ran. We will change this behaviour ASAP such that you can use just 1 queue for your use case.

Hi, I found the problem using the example Martin gave. Apparently you cannot use pipe.start_locally() at all when trying to clone the task and work completely remote (I thought it would treat the agent as local instead when I send it to a queue). It works with the combination of pipe.set_default_execution_queue('agent') and pipe.start(queue = 'agent2(EC2)') . However, must I really have two clearml-agents for complete automation? To the best of my knowledge, setting both the function above to the same queue will just cause an infinite queue. Is there no way to use only one worker for everything like start_locally(run_pipeline_steps_locally=True) ? For example, initially I thought if I use Task.enqueue(task = clone_task.id, queue_name= 'agent2(EC2)') (cloning pipeline) and start_locally(run_pipeline_steps_locally=True) (pipeline file), clearml will treat the agent2(EC2) as local instead.

Hi LittleShrimp86 ! Looks like something is broken. We are looking into it

This is very odd...
LittleShrimp86 is this example working for you?
https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_tasks.py

Yes the configuration and stages (DAG visualization) were there right until the agent finish cloning the environment. Then it goes missing.

Wait. are you saying it is disappearing ? meaning when you cloned the Pipeline (i.e. in draft mode) the configuration was there, then the configuration disappeared ?

I dont know why 😢 . Updated to 1.7.2, was staring at the configuration object, there was stuff in it until it reached "Starting Task Execution:" in the logs. It went missing after

The configuration tab -> configuration objects -> pipeline is empty

That's the reason it is doing nothing 😞
How come it is empty if you Cloned the local one?

Working on the 1.7.2 right now

The configuration tab -> configuration objects -> pipeline is empty

BTW: Can you also please test with the latest clearml version , 1.7.2

LittleShrimp86 what do you have in the Configuration Tab of the Cloned Pipeline?
(I think that it has empty configuration -> which means empty DAG, so it does nothing and leaves)

Original one

This is the cloned one

LittleShrimp86 can you post the full log of the pipeline? (something is odd here)

Still the same problem 😕 . I used pipe.set_default_execution_queue('agent') and pipe.start_locally() for my pipeline.py. When cloning, I enqueue the pipeline to another agent: Task.enqueue(task = clone_task.id, queue_name= 'agent(EC2)')

Now I just need to wait for this to finish and clone it later 🤞

Seems to be working, its in the first stage 😮

All right testing with:
pipe.set_default_execution_queue('agent') pipe.start_locally() pipe.start(queue= 'agent')

I mean test with:
pipe.start_locally(run_pipeline_steps_locally=False)This actually creates the steps as Tasks and launches them on remote machines

Let me test it out. I thought if I run it remotely with pipe.start_locally(run_pipeline_steps_locally=True) , the local becomes remote instead

No worries, you should probably change it to pipe.start(queue= 'queue') not start locally
s it working when you are calling it with start locally ?

Sorry for the comments haha. Trying to note down whatever I learn as much as I can

` from clearml import PipelineController

pipe = PipelineController(name="clearmlsample_pipeline",
project="clearmlsample",
version="1.0.0")

pipe.add_parameter('seed', 2222, description='random seed to standardize randomness')
pipe.add_parameter('n_trials', 10, description='trials to run during optimization')

pipe.add_step(
name='get_data', # can be named anything
# connect pipeline to task (obtain data from Task.init in python files)
base_task_project='clearmlsample', # project name
base_task_name='get data & preprocess' # task name
)

pipe.add_step(
name='train_and_tune', # name
base_task_project='clearmlsample', # project name
base_task_name='training and tuning', # task name
# connect it to previous task using name not base_task_name
# step cannot run unless parents finish
parents=['get_data'],
# use 'General/parameter' to override parameter from py file
# give value or take value from pipe.add_parameter using '${pipeline.parameter}'
#

# don need any special command to get data from Dataset.create() from previous task
# just use the Dataset.get to get latest version
# maybe easier to store everything except (raw data) in artificats instead
parameter_override={'General/seed': '${pipeline.seed}',
                    'General/n_trials': '${pipeline.n_trials}'}

)

pipe.add_step(
name='evaluate', # name
base_task_project='clearmlsample', # project name
base_task_name='evaluating', # task name
# connect it to previous task use name not base_task_name
parents=['train_and_tune'],
parameter_override={'General/seed': '${pipeline.seed}',
# get task id from previous step to get models
'General/train_task_id': '${train_and_tune.id}'}
)

select default worker to run steps

pipe.set_default_execution_queue('MonashPC')

start the pipeline logic

run this to run EVERYTHING locally

pipe.start_locally(run_pipeline_steps_locally=True)

run this to run logic locally but steps remotely

pipe.start_locally()

run this to run logic remotely

do not start pipeline with same worker as set_default_exceution_queue

e.g. pipe.start(queue='MonashPC')

it will cause the steps to be queued forever because it is occuped by the pipeline logic

pipe.start(queue= 'queue')

print('done') `

Can you share the pipeline?

I also tried cloning an individual task, That surprisingly works. Not sure why my pipeline doesn't.

Yup, the pipeline stops instantly with "Launching the next 0 steps".

LittleShrimp86 did you try to run the pipeline form the UI on remote machines (i.e. with the agents)? Did that work?

Yes, similar but via Github Actions for automation. Just wanted to know if there is an easier way to connect to clearml instead of creating a new workflow for any CI/CD purpose.

Aside from that, I tried cloning my task (pipeline) and enqueuing it to a clearml-agent.
filter = {'status': ['published'], 'order_by': ['-last_update'], 'type': ['controller']} pipeline_task = Task.get_tasks(project_name='clearmlsample/clearmlsample_pipeline/.pipeline', task_filter = filter) clone_task = Task.clone(source_task=pipeline_task[0], name = "Clone pipeline") Task.enqueue(task = clone_task.id, queue_name= 'agent')I already ran the pipeline a few times locally which works fine. However when I try cloning it, the stages seem to go missing and the pipeline ends instantly.

Hi LittleShrimp86

just to login into your clearml app (demo or server) so I can run python files related to clearml.

I think this amounts to creating a Task and enqueueing it, am I understanding correctly ?