Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Hi, I'Ve Three Questions Regarding Clearml Pipelines.

Hi, I've three questions regarding clearml pipelines.

  • can I check when we use a clearml pipeline and data get transferred from stage to stage, do the data get serialised or pickled? If it does, could you please point me to the relevant codes on GitHub? Some of our folks found that they can't transfer files from stage to stage. In this case pth files.
  • Second is, for k8s, can we give an option to run a pipeline instance on a single node, whichever it's assigned to. Or simply put, the stages on the same pipeline runs on same node? Reason is we want to improve the stage to stage transfer performance.
  • For pipelines, can we somehow create a dynamic PV on the fly that allows all the stages to share a common temp location? This is useful in scenarios where we need to pull ClearMl-datasets and we can reduce the data pull to just once.
Posted 9 months ago
Votes Newest


Hi @<1523701504827985920:profile|SubstantialElk6> !
Regarding 1: pth files get pickled.
The flow is like this:

  • The step is created by the controller by writing some code to a file and running that file in python
  • The following line is ran in the step when returning values: None
  • This is eventually ran: None
  • The file will get pickled here: NoneNote that we have a this argument: None . So if your files can’t be pickled, you can serialize them some other way (maybe use dill None )
Posted 9 months ago
1 Answer
9 months ago
9 months ago