Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, Playing Around With Hp Optimisation, And I Notice In The Hyperparameteroptimizer Class Itself, The

Hi all, playing around with HP optimisation, and I notice in the HyperParameterOptimizer class itself, the objective_metric_sign is set to min by default. Can someone elaborate on min vs global_min? For example, Im trying to optimise a simple RF regressor using MAE, so I selected min, only to have the example “WOOT WOOT we broke the record” print out frequently and not actually minimise. Im a bit confused as to what I might have missed or misconfigured. Perhaps the job IDs arent changing so it all jobs simple print WOOT WOOT? Perhaps min instead of global min just means if its better than the last iteration, regardless of if its the best, the job gets assigned top_performance_job_id?

` :param str objective_metric_sign: The objective to maximize / minimize.

        The values are:

        - ``min`` - Minimize the last reported value for the specified title/series scalar.
        - ``max`` - Maximize the last reported value for the specified title/series scalar.
        - ``min_global`` - Minimize the min value of *all* reported values for the specific title/series scalar.
        - ``max_global`` - Maximize the max value of *all* reported values for the specific title/series scalar. `

WOOT WOOT we broke the record! Objective reached 16.159396398659123 WOOT WOOT we broke the record! Objective reached 17.17549404054404 WOOT WOOT we broke the record! Objective reached 17.071016994817196 WOOT WOOT we broke the record! Objective reached 17.14302934610711

  
  
Posted 3 years ago
pic
More
LudicrousParrot69
0 × 1
Report
Votes Newest

Answers 8

Hi LudicrousParrot69
I guess you are right this is not trivial distinction:
min: means we are looking for the the minimum value of a specific scalar. meaning 1.0, 0.5, 1.3 -> the optimizer will get these direct values and will optimize based on that
global min: means the optimizer is getting the minimum values of the specific scalar. With the same example: 1.0, 0.5, 1.3 -> the HPO optimizer gets 1.0, 0.5, 0.5
The same holds for max/global_max , make sense ?

  
  
Posted 3 years ago
pic
More
AgitatedDove14
0 × 1
Report

Alas no, apologies. Are you saying that in the global_min case, if a trial returns an MAE of 1.3, but the previous trial got an MAE of 0.5, the optimiser gets told that the MAE of the latest model is 0.5 instead of the truth?

  
  
Posted 3 years ago
pic
More
LudicrousParrot69
0 × 1
Report

Correct, which makes sense if you have a stochastic process and you are looking for the best model snapshot. That said I guess the default use case would be min/max (and not the global variant)

  
  
Posted 3 years ago
pic
More
AgitatedDove14
0 × 1
Report

Okay, that makes sense then. Whats still got me scratching my head is the examples printing out the WOOT WOOT for breaking the record, despite it clearly not. Hmm.

  
  
Posted 3 years ago
pic
More
LudicrousParrot69
0 × 1
Report

Hmmm:

WOOT WOOT we broke the record! Objective reached 17.071016994817196
WOOT WOOT we broke the record! Objective reached 17.14302934610711

These two seems strange, let me look into it

  
  
Posted 3 years ago
pic
More
AgitatedDove14
0 × 1
Report

Found it, definitely a bug in the callback, it has not effect on the HPO process itself

  
  
Posted 3 years ago
pic
More
AgitatedDove14
0 × 1
Report

Phew! If I find any bugs or potential issues in the doco/comments too, where would be the best place to send that so I dont spam slack if I find tiny issues. Github issues / DM / a specific slack channel?

  
  
Posted 3 years ago
pic
More
LudicrousParrot69
0 × 1
Report

Bugs, definitely GitHub, this is the easiest to track.
Documentation, if these are small issues, Slack is fine, otherwise, GitHub issue.
Regrading the documentation, we are working on another iteration of improvement, but if you find inaccuracies/broken links please report 🙂

  
  
Posted 3 years ago
pic
More
AgitatedDove14
0 × 1
Report
Write your answer
941 Views
8 Answers
3 years ago
one year ago
Tags
clearml