Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Community! I Have Difficulty Using Clearml Pipeline. I Am Writing The Code Using The Pipeline Decorator, But The Pipeline Does Not Work With The Following Error When Specifying The Docker Image As A Argument Of The Decorator. How Should I Solve It?

Hi community! I have difficulty using Clearml pipeline. I am writing the code using the pipeline decorator, but the pipeline does not work with the following error when specifying the docker image as a argument of the decorator. How should I solve it?

Setting pipeline controller Task as failed (due to failed steps) !
Traceback (most recent call last):
  File "/root/.clearml/venvs-builds/3.8/code/copy_pipeline.py", line 255, in <module>
    main(path1, threshold, num, path2)
  File "/root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/clearml/automation/controller.py", line 4391, in internal_decorator
    raise triggered_exception
  File "/root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/clearml/automation/controller.py", line 4368, in internal_decorator
    LazyEvalWrapper.trigger_all_remote_references()
  File "/root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/clearml/utilities/proxy_object.py", line 399, in trigger_all_remote_references
    func()
  File "/root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/clearml/automation/controller.py", line 4033, in results_reference
    raise ValueError(
ValueError: Pipeline step "training", Task ID=631f674c8ca248219e23cccc98f26482 failed

This is part of my pipeline code. Thanks!!

@PipelineDecorator.component(return_values=['int'], cache=False,
                             task_type='training',
                             docker="ultralytics/yolov5",
                             parents=['preprocessing'])
def training(yaml_path, yolo_path):
    print("training end")
    return 1
  
  
Posted 5 months ago
Votes Newest

Answers 14


Where is the log of the Task that is created for the component?
The error you are getting is the Pipeline itself throwing error because the component Task failed, the question is why did the component Task failed

  
  
Posted 5 months ago

I wish I had a more detailed log file why task component failed, but that's all log files are.
The one line log file is the log of the component task.

  
  
Posted 5 months ago

Are you saying you have a single line in the console output of the component Task?

  
  
Posted 5 months ago

Just to make sure, the first two steps are working ?
Maybe it has to do with the fact the "training" step specifies a docker image, could you try to remove it and check?
BTW: A few pointers
The return_values is used to specify multiple returned objects stored individually, not the type of the object. If there is a single object, no need to specify
The parents argument is optional, the pipeline components optimizes execution based on inputs, for example in your code, all pipeline components will be running in parallel, because they do not rely on the return value of one another, unless of course if you specify " parents " 🙂

  
  
Posted 5 months ago

Hmm this is odd, could you provide the pipeline code maybe?

  
  
Posted 5 months ago

Yes, here it is. Very simple code. Thanks 🙂

  
  
Posted 5 months ago

Yes, that's what I'm saying.

  
  
Posted 5 months ago

Thanks @<1634001106403069952:profile|DefeatedMole42>
A follow up, (1) how are you spinning the agent ? (2) could it be the docker image "ultralytics/yolov5" does not have Bash as entry point ?
you can force that with

@PipelineDecorator.component(return_values=['int'], cache=False,
                             task_type='training',
                             docker="ultralytics/yolov5",
                             docker_args="--entrypoint /bin/bash",
                             parents=['preprocessing'])
  
  
Posted 5 months ago

I tested it with a simpler code to make it more clear!
I've attached the code and log files that I used.
It only fails if I use the docker image.. 😂
Thank you so much for pointing out the argument. I think I can use it better!!! 👍
image

  
  
Posted 5 months ago

Hi, @<1523701205467926528:profile|AgitatedDove14>
I am attaching two log files.
One is the log file of the task that ran the pipeline code, and the other is the log file of the component.

  
  
Posted 5 months ago

Hi @<1634001106403069952:profile|DefeatedMole42>
This points to the pipeline component failing to execute (i.e. the Task of the component Failed)
Can you send the log of that Task?

  
  
Posted 5 months ago

How come the second one is one line?

  
  
Posted 5 months ago

I'm running the main pipeline code remotely(not using run_locally).
The component in question seems to fail almost immediately after pipeline execution because of value error.
If I put docker argument in the decorator, value error appears and becomes failure..

  
  
Posted 5 months ago

How are you running it?

  
  
Posted 5 months ago
330 Views
14 Answers
5 months ago
5 months ago
Tags
Similar posts