Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi I Am Having Issue With Pipeline. Even My Every Task Of Pipeline Is Completed, My Core Pipeline Controller Is Still Running. I Have Added Pipe.Wait() And Pipe.Stop() As Well. Attached Pic Shows The Pipeline In Running State, And Tasks In Completed Stat

Hi
I am having issue with pipeline.
Even my every task of pipeline is completed, my core pipeline controller is still running.

I have added pipe.wait() and pipe.stop() as well.
Attached pic shows the pipeline in running state, and tasks in completed state
image

  
  
Posted 8 months ago
Votes Newest

Answers 16


Hi @<1585078763312386048:profile|ArrogantButterfly10> , does the controller stay indefinitely in the running state?

  
  
Posted 8 months ago

yes

  
  
Posted 8 months ago

clearml==1.9.1
clearml-agent==1.5.2

I am not self hosting the server, using the one provided by clearml side

  
  
Posted 8 months ago

What version of clearml , clearml-agent & server are you using?

  
  
Posted 8 months ago

Can you update the clearml version to latest (1.11.1) and see whether the issue is fixed?

  
  
Posted 8 months ago

sure, I ll check and update

  
  
Posted 8 months ago

from clearml import Task
from clearml.automation import PipelineController
pipe = PipelineController(name='PIPE_TEST_3',project='PIPE_TEST_3',version="0.0.1",add_pipeline_tags=False)

pipe.add_parameter("url",
" None ",
"dataset_url"
)

pipe.set_default_execution_queue('services')

pipe.add_step(name="stage_data",
base_task_project="PIPE_TEST_3",
base_task_name="Pipeline step 1 dataset artifact",
parameter_override={"General/dataset_url": "${pipeline.url}"})

pipe.add_step(
name="stage_process",
parents=["stage_data"],
base_task_project="PIPE_TEST_3",
base_task_name="Pipeline step 2 process dataset",
parameter_override={
"General/dataset_url": "${stage_data.artifacts.dataset.url}",
"General/test_size": 0.25,
}
)

pipe.add_step(
name="stage_train",
parents=["stage_process"],
base_task_project="PIPE_TEST_3",
base_task_name="Pipeline step 3 process dataset",
parameter_override={"General/dataset_task_id": "${stage_process.id}"},

)

# pipe.start_locally()
pipe.start(queue='services')

  
  
Posted 8 months ago

Hey @<1537605940121964544:profile|EnthusiasticShrimp49> I updated clearml but now the issue is my pipeline is stuck here.
Previously it was working fine till the above mentioned issue and I made no change except the mentioned.
image

  
  
Posted 8 months ago

Can you please attach the code for the pipeline?

  
  
Posted 8 months ago

Ignore default, I am trying to run with another docker, but it is also stuck as same
image

  
  
Posted 8 months ago

And how many agents do you have listening on the “services“ queue?

  
  
Posted 8 months ago

one

  
  
Posted 8 months ago

I see you want to use the services queue for both the pipeline controller and pipeline steps, but you have only one worker/agent listening to this queue. In this case you need at least 2 agents listening to the services queue. Try spawning an additional agent that listens to this queue and let me know how it goes .

  
  
Posted 8 months ago

It ran, thanks.. but that original problem persisits. Pipeline is running once all the tasks completed.

  
  
Posted 8 months ago

@<1537605940121964544:profile|EnthusiasticShrimp49> is this a code issue or some bug?

  
  
Posted 8 months ago

image

  
  
Posted 8 months ago
463 Views
16 Answers
8 months ago
8 months ago
Tags