Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi I Am Having Issue With Pipeline. Even My Every Task Of Pipeline Is Completed, My Core Pipeline Controller Is Still Running. I Have Added Pipe.Wait() And Pipe.Stop() As Well. Attached Pic Shows The Pipeline In Running State, And Tasks In Completed Stat

Hi
I am having issue with pipeline.
Even my every task of pipeline is completed, my core pipeline controller is still running.

I have added pipe.wait() and pipe.stop() as well.
Attached pic shows the pipeline in running state, and tasks in completed state
image

  
  
Posted one year ago
Votes Newest

Answers 16


Ignore default, I am trying to run with another docker, but it is also stuck as same
image

  
  
Posted one year ago

@<1537605940121964544:profile|EnthusiasticShrimp49> is this a code issue or some bug?

  
  
Posted one year ago

from clearml import Task
from clearml.automation import PipelineController
pipe = PipelineController(name='PIPE_TEST_3',project='PIPE_TEST_3',version="0.0.1",add_pipeline_tags=False)

pipe.add_parameter("url",
" None ",
"dataset_url"
)

pipe.set_default_execution_queue('services')

pipe.add_step(name="stage_data",
base_task_project="PIPE_TEST_3",
base_task_name="Pipeline step 1 dataset artifact",
parameter_override={"General/dataset_url": "${pipeline.url}"})

pipe.add_step(
name="stage_process",
parents=["stage_data"],
base_task_project="PIPE_TEST_3",
base_task_name="Pipeline step 2 process dataset",
parameter_override={
"General/dataset_url": "${stage_data.artifacts.dataset.url}",
"General/test_size": 0.25,
}
)

pipe.add_step(
name="stage_train",
parents=["stage_process"],
base_task_project="PIPE_TEST_3",
base_task_name="Pipeline step 3 process dataset",
parameter_override={"General/dataset_task_id": "${stage_process.id}"},

)

# pipe.start_locally()
pipe.start(queue='services')

  
  
Posted one year ago

image

  
  
Posted one year ago

Hey @<1537605940121964544:profile|EnthusiasticShrimp49> I updated clearml but now the issue is my pipeline is stuck here.
Previously it was working fine till the above mentioned issue and I made no change except the mentioned.
image

  
  
Posted one year ago

Can you please attach the code for the pipeline?

  
  
Posted one year ago

sure, I ll check and update

  
  
Posted one year ago

And how many agents do you have listening on the “services“ queue?

  
  
Posted one year ago

I see you want to use the services queue for both the pipeline controller and pipeline steps, but you have only one worker/agent listening to this queue. In this case you need at least 2 agents listening to the services queue. Try spawning an additional agent that listens to this queue and let me know how it goes .

  
  
Posted one year ago

one

  
  
Posted one year ago

clearml==1.9.1
clearml-agent==1.5.2

I am not self hosting the server, using the one provided by clearml side

  
  
Posted one year ago

Can you update the clearml version to latest (1.11.1) and see whether the issue is fixed?

  
  
Posted one year ago

yes

  
  
Posted one year ago

Hi @<1585078763312386048:profile|ArrogantButterfly10> , does the controller stay indefinitely in the running state?

  
  
Posted one year ago

What version of clearml , clearml-agent & server are you using?

  
  
Posted one year ago

It ran, thanks.. but that original problem persisits. Pipeline is running once all the tasks completed.

  
  
Posted one year ago
924 Views
16 Answers
one year ago
one year ago
Tags