Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello All, I’M An Ml Engineer Looking To Transition Our Company To A New Mlops System. Many Of Our Projects Are Currently Built Around Hydra And I’M Attempting To See What I Would Need To Do To Integrate Clearml Into Our Workflow. I’M Fully Aware That You

Hello all,
I’m an ML engineer looking to transition our company to a new MLops system. Many of our projects are currently built around Hydra and I’m attempting to see what I would need to do to integrate ClearML into our workflow. I’m fully aware that you can pass _allow_omegaconf_edit_ : True and modify the Hydra config, which does all me to modify the config, however this is a bit lacking on two fronts, one of which I might have a solution for but the other has to do with nested group configs

None
Basically, you can have group configs in a main config that allows you to specify a “group” of parameters. For example, in my main config I can have the following:

defaults:
- _self_
- datamodule: fraud
- model: gnn
- callbacks: default
- logger: null
- trainer: default
- paths: default
- extras: default
- hydra: default

Each of these defaults are config groups, a folder containing several groups depending on the run conditions.
├── configs <- Hydra configuration files
│ ├── callbacks <- Callbacks configs
│ ├── datamodule <- Datamodule configs
│ ├── debug <- Debugging configs
│ ├── experiment <- Experiment configs
│ ├── extras <- Extra utilities configs
│ ├── hparams_search <- Hyperparameter search configs
│ ├── hydra <- Hydra configs
│ ├── local <- Local configs
│ ├── logger <- Logger configs
│ ├── model <- Model configs
│ ├── paths <- Project paths configs
│ ├── trainer <- Trainer configs
│ │
│ ├── eval.yaml <- Main config for evaluation
│ └── train.yaml <- Main config for training

It would be nice to be able to change the group configs from ClearML. It could be done by completely copying the content of the group config parameters into the hydra config that is saved by ClearML, but this defeats the purpose of having the group configs. For example: if I want to change which model the code instantiates, I can override it at runtime:
python src/train.py model=gnn3

It appears that ClearML captures these command line arguments under args:
overrides ['model=gnn3']

However, if I clone the experiment and modify this argument it doesn’t change anything. Is there a way to modify the overrides that are passed at runtime in a way that Hydra can capture them? I’m currently using hydra-core 1.3.0.

  
  
Posted one year ago
Votes Newest

Answers 3


I might have found the answer. I'll reply if it works as expected.

  
  
Posted one year ago

The answer is simple but also not completely obvious to someone new to the platform. So you can inject new command line args that hydra will recognize. This is what the Hydra section of args is for. However, if you enable _allow_omegaconf_edit_: True , I think ClearML will “inject” the OmegaConf saved under the configuration object of the prior run, overwriting the overrides. I’ll experiment with this behavior a bit more to be sure.

  
  
Posted one year ago

Hi @<1545216070686609408:profile|EnthusiasticCow4> !

So you can inject new command line args that hydra will recognize.

This is true.

However, if you enable _allow_omegaconf_edit_: True, I think ClearML will "inject" the OmegaConf saved under the configuration object of the prior run, overwriting the overrides

This is also true.

  
  
Posted one year ago
1K Views
3 Answers
one year ago
one year ago
Tags