
Reputation
Badges 1
17 × Eureka!Still the same problem 😕 . I used pipe.set_default_execution_queue('agent')
and pipe.start_locally()
for my pipeline.py. When cloning, I enqueue the pipeline to another agent: Task.enqueue(task = clone_task.id, queue_name= 'agent(EC2)')
Sorry for the comments haha. Trying to note down whatever I learn as much as I can
Yes the configuration and stages (DAG visualization) were there right until the agent finish cloning the environment. Then it goes missing.
The configuration tab -> configuration objects -> pipeline is empty
` from clearml import PipelineController
pipe = PipelineController(name="clearmlsample_pipeline",
project="clearmlsample",
version="1.0.0")
pipe.add_parameter('seed', 2222, description='random seed to standardize randomness')
pipe.add_parameter('n_trials', 10, description='trials to run during optimization')
pipe.add_step(
name='get_data', # can be named anything
# connect pipeline to task (obtain data from Task.init in python fi...
Now I just need to wait for this to finish and clone it later 🤞
Yes, similar but via Github Actions for automation. Just wanted to know if there is an easier way to connect to clearml instead of creating a new workflow for any CI/CD purpose.
Aside from that, I tried cloning my task (pipeline) and enqueuing it to a clearml-agent.
` filter = {'status': ['published'], 'order_by': ['-last_update'], 'type': ['controller']}
pipeline_task = Task.get_tasks(project_name='clearmlsample/clearmlsample_pipeline/.pipeline',
task_filter = f...
Let me test it out. I thought if I run it remotely with pipe.start_locally(run_pipeline_steps_locally=True)
, the local becomes remote instead
Hi, I found the problem using the example Martin gave. Apparently you cannot use pipe.start_locally()
at all when trying to clone the task and work completely remote (I thought it would treat the agent as local instead when I send it to a queue). It works with the combination of pipe.set_default_execution_queue('agent')
and pipe.start(queue = 'agent2(EC2)')
. However, must I really have two clearml-agents for complete automation? To the best of my knowledge, setting both ...
Yup, the pipeline stops instantly with "Launching the next 0 steps".
All right testing with:pipe.set_default_execution_queue('agent') pipe.start_locally() pipe.start(queue= 'agent')
I dont know why 😢 . Updated to 1.7.2, was staring at the configuration object, there was stuff in it until it reached "Starting Task Execution:" in the logs. It went missing after
Seems to be working, its in the first stage 😮
I also tried cloning an individual task, That surprisingly works. Not sure why my pipeline doesn't.