Hello all,
I’m an ML engineer looking to transition our company to a new MLops system. Many of our projects are currently built around Hydra and I’m attempting to see what I would need to do to integrate ClearML into our workflow. I’m fully aware that you can pass _allow_omegaconf_edit_ : True and modify the Hydra config, which does all me to modify the config, however this is a bit lacking on two fronts, one of which I might have a solution for but the other has to do with nested group configs
None
Basically, you can have group configs in a main config that allows you to specify a “group” of parameters. For example, in my main config I can have the following:
defaults:
- _self_
- datamodule: fraud
- model: gnn
- callbacks: default
- logger: null
- trainer: default
- paths: default
- extras: default
- hydra: default
Each of these defaults are config groups, a folder containing several groups depending on the run conditions.
├── configs <- Hydra configuration files
│ ├── callbacks <- Callbacks configs
│ ├── datamodule <- Datamodule configs
│ ├── debug <- Debugging configs
│ ├── experiment <- Experiment configs
│ ├── extras <- Extra utilities configs
│ ├── hparams_search <- Hyperparameter search configs
│ ├── hydra <- Hydra configs
│ ├── local <- Local configs
│ ├── logger <- Logger configs
│ ├── model <- Model configs
│ ├── paths <- Project paths configs
│ ├── trainer <- Trainer configs
│ │
│ ├── eval.yaml <- Main config for evaluation
│ └── train.yaml <- Main config for training
It would be nice to be able to change the group configs from ClearML. It could be done by completely copying the content of the group config parameters into the hydra config that is saved by ClearML, but this defeats the purpose of having the group configs. For example: if I want to change which model the code instantiates, I can override it at runtime:
python src/train.py model=gnn3
It appears that ClearML captures these command line arguments under args:
overrides ['model=gnn3']
However, if I clone the experiment and modify this argument it doesn’t change anything. Is there a way to modify the overrides that are passed at runtime in a way that Hydra can capture them? I’m currently using hydra-core 1.3.0.
