Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello, I Am Currently Learning How To Build Pipelines With Clearml. I'Ve Created A Pipeline That Has Five Steps, In Which Each Step Depends On The Previous One (Step 2 Depends On Step 1, Step 3 Depends On Step 2, Etc.). However, My Step 4 Depends On Step

Hello, I am currently learning how to build pipelines with clearml.
I've created a pipeline that has five steps, in which each step depends on the previous one (step 2 depends on step 1, step 3 depends on step 2, etc.). However, my step 4 depends on step 2 and step 3, and step 5 depends on step 2 and step 4. I specifically wrote: parents=['step_two', 'step_three'] for step 4, and parents=['step_two', 'step_four'] for step 5.
My pipeline has completed successfully both locally and remotely, however, in the UI my pipeline has only sequational structure (I've attached screenshot).

Could someone please explain why the pipeline structure doesn't have links ( step 2 and step 4 ) and ( step 2 and step 5 )?

I've checked pipeline's configuration object. Each of these steps has resolved parent tasks correctly, like I have written in the code, but the UI structure is different and I wonder why.

I've built the pipeline from functions. If necessary I can attach the code of building the pipeline
image

  
  
Posted one year ago
Votes Newest

Answers 5


Pipeline's configuration object:

{
"step_one": {
"base_task_id": null,
"queue": "test_pipeline",
"parents": [],
"timeout": null,
"parameters": {},
"configurations": {},
"task_overrides": {},
"executed": "85259fca41e74e888827b279f0dd80ad",
"status": "completed",
"clone_task": false,
"job_type": "custom",
"job_started": 1677573232.5546257,
"job_ended": 1677574185.0023673,
"job_code_section": null,
"skip_job": false,
"continue_on_fail": false,
"cache_executed_step": false,
"return_artifacts": [
"data",
"targets"
],
"monitor_metrics": null,
"monitor_artifacts": null,
"monitor_models": null,
"job_id": "85259fca41e74e888827b279f0dd80ad"
},
"step_two": {
"base_task_id": null,
"queue": "test_pipeline",
"parents": [
"step_one"
],
"timeout": null,
"parameters": {
"kwargs_artifacts/X": "${step_one.id}.data",
"kwargs_artifacts/y": "${step_one.id}.targets"
},
"configurations": {},
"task_overrides": {},
"executed": "9525f8ee68e146bc864d54b118fda35b",
"status": "completed",
"clone_task": false,
"job_type": "data_processing",
"job_started": 1677574184.3924935,
"job_ended": 1677574225.0869856,
"job_code_section": null,
"skip_job": false,
"continue_on_fail": false,
"cache_executed_step": false,
"return_artifacts": [
"X_train",
"X_test",
"y_train",
"y_test"
],
"monitor_metrics": null,
"monitor_artifacts": null,
"monitor_models": null,
"job_id": "9525f8ee68e146bc864d54b118fda35b"
},
"step_three": {
"base_task_id": null,
"queue": "test_pipeline",
"parents": [
"step_two"
],
"timeout": null,
"parameters": {
"kwargs/rf_hyper_params": "${pipeline.rf_params}",
"kwargs_artifacts/X_train": "${step_two.id}.X_train",
"kwargs_artifacts/y_train": "${step_two.id}.y_train"
},
"configurations": {},
"task_overrides": {},
"executed": "1024945bccf9467d8155d61060475024",
"status": "completed",
"clone_task": false,
"job_type": "training",
"job_started": 1677574224.3591328,
"job_ended": 1677574265.5506012,
"job_code_section": null,
"skip_job": false,
"continue_on_fail": false,
"cache_executed_step": false,
"return_artifacts": [
"model"
],
"monitor_metrics": null,
"monitor_artifacts": null,
"monitor_models": null,
"job_id": "1024945bccf9467d8155d61060475024"
},
"step_four": {
"base_task_id": null,
"queue": "test_pipeline",
"parents": [
"step_two",
"step_three"
],
"timeout": null,
"parameters": {
"kwargs_artifacts/model": "${step_three.id}.model",
"kwargs_artifacts/X_test": "${step_two.id}.X_test"
},
"configurations": {},
"task_overrides": {},
"executed": "23fca63d4cd540baa5f30eddc9f9c5f9",
"status": "completed",
"clone_task": false,
"job_type": "testing",
"job_started": 1677574264.5031798,
"job_ended": 1677574294.0008907,
"job_code_section": null,
"skip_job": false,
"continue_on_fail": false,
"cache_executed_step": false,
"return_artifacts": [
"prediction"
],
"monitor_metrics": null,
"monitor_artifacts": null,
"monitor_models": null,
"job_id": "23fca63d4cd540baa5f30eddc9f9c5f9"
},
"step_five": {
"base_task_id": null,
"queue": "test_pipeline",
"parents": [
"step_two",
"step_four"
],
"timeout": null,
"parameters": {
"kwargs_artifacts/y_pred": "${step_four.id}.prediction",
"kwargs_artifacts/y_true": "${step_two.id}.y_test"
},
"configurations": {},
"task_overrides": {},
"executed": "2ee0f781e09e4e1a96666a92de381f70",
"status": "completed",
"clone_task": false,
"job_type": "qc",
"job_started": 1677574292.9079442,
"job_ended": 1677574332.6114872,
"job_code_section": null,
"skip_job": false,
"continue_on_fail": false,
"cache_executed_step": false,
"return_artifacts": [
"acc_value"
],
"monitor_metrics": null,
"monitor_artifacts": null,
"monitor_models": null,
"job_id": "2ee0f781e09e4e1a96666a92de381f70"
}
}

  
  
Posted one year ago

Hi @<1523704757024198656:profile|MysteriousWalrus11> , do you have a snippet that reproduces this? When running locally and remotely the way the pipeline shows up is the same?

  
  
Posted one year ago

Hi @<1523704757024198656:profile|MysteriousWalrus11>

"parents": [

  "step_two", 
  "step_four" 
], 

Seems like step 5 depends on steps 2+4 , how did you create it? what did the console say ?
Could it be your not actually passing any output from step3 ? how is it dependent on it ?

  
  
Posted one year ago

I understood what was a problem in my case

Look closely to the screenshot. Right now "step_five" is selected (it has "step_two" and "step_four" as parents). There is actually an arrow from "step_two" to "step_five" (it is slightly lighter than grey arrow). It just layers on another arrow, so it's a "graph's edges location"/"GUI" problem
image

  
  
Posted one year ago

Okay, I'll pass to front-end, see what they can do about it.

  
  
Posted one year ago