Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, We Have A Workflow Which Goes Over List Of Directories And Processes All Movies From Them. "Process" - Means Run Certain Detection Algorithms On Each Movie Frame. We Built Clearml Task From This Workflow, And Created Hpo Application Based On This Task

Hi,
We have a workflow which goes over list of directories and processes all movies from them. "Process" - means run certain detection algorithms on each movie frame. We built ClearML task from this workflow, and created HPO application based on this task. Obviously there are many repetitive steps in this flow - i.e. most of our hyperparameters don't affect all detection blocks.
Our question is: does ClearML provide a way to minimize a time for this HPO app? We through about ClearML Pipelines, but we see no way to use them without heavy refactoring.

  
  
Posted one year ago
Votes Newest

Answers


Unfortunately, ClearML HPO does not "know" what is inside the task it is optimizing. It is like that by design, so that you can run HPO with no code changes inside the experiment. That said, this also limits us in not being able to "smartly" optimize.

However, is there a way you could use caching within your code itself? Such as using functools' LRU cache? This is built-in in python and will cache function return values if it's ever called again with the same input arguments.

There also seem to be ways to cache to a local disk folder (check here second answer). You could run it locally first and then e.g. save the cache as a clearml dataset, then pull it for each worker. Or: you could have a central folder somewhere on a network drive that is accessible by all your workers, and have them use the same cache folder?

  
  
Posted one year ago
974 Views
1 Answer
one year ago
one year ago
Tags