Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Given I Want To Run A Task In A Pipeline Using A Base Task Id. One Of My Steps Just Finds The Latest Model To Use. I Want The Task To Output The Id, And The Next Step To Use It. How Would I Go About Doing This?

Given I want to run a task in a pipeline using a base task id. One of my steps just finds the latest model to use. I want the task to output the id, and the next step to use it. How would I go about doing this?

  
  
Posted 2 years ago
Votes Newest

Answers 10


Thus I wanted to pass the model id from the prior step to the next one.

  
  
Posted 2 years ago

but I can't seem to figure out a way to do something similar using a task in add_step

VexedCat68 With "add_step" it assumes the Task you are adding is self contained (i.e. there is no "return object" to serialize), this means you can only add arguments, or use the artifacts the Task (i.e. step) will recreate, obviously you knowing in advance what the step creates. Make sense ?

  
  
Posted 2 years ago

VexedCat68 yes 🙂 you can also pass the parent folder and it will zip the entire subfolders into a single artifact

  
  
Posted 2 years ago

I use that model in another step.

  
  
Posted 2 years ago

Well yeah, you can say that. In add function step, I pass in a function which returns something. And since I've written the name of the returned parameter in add_function_step, I can use it, but I can't seem to figure out a way to do something similar using a task in add_step

  
  
Posted 2 years ago

Hi VexedCat68

One of my steps just finds the latest model to use. I want the task to output the id, and the next step to use it. How would I go about doing this?

When you say "I want the task to output the id" do you mean to pass t to the next step:
Something like this one:
https://github.com/allegroai/clearml/blob/c226a748066daa3c62eddc6e378fa6f5bae879a1/clearml/automation/controller.py#L224

  
  
Posted 2 years ago

is this the correct way to upload an artifact?

checkpoint.split('.')[0] is the name that I want it assigned and the second argument is the path to the file.

  
  
Posted 2 years ago

I got to that conclusion I think yeah. Basically, can access them as artifacts.

  
  
Posted 2 years ago

Its a simple DAG pipeline.
I have a step, at which I want to run a task which finds the model I need.

  
  
Posted 2 years ago

Can you give a bit more info of how you want the pipeline built and where you want to insert/extract the task id? Also how is the model related? Is it the start of the pipeline?

  
  
Posted 2 years ago