Hi again CostlyOstrich36 ,
I just wanted to share what ended up working for me. Basically I worked it out both for Hydra (thanks CurvedHedgehog15 ) and for PytorchLightningCLI.
So, for PL-CLI, I used this construct so we don't have to modify our training scripts based on our experiment tracker
` from pytorch_lightning.utilities.cli import LightningCLI
from clearml import Task
class MyCLI(LightningCLI):
def before_instantiate_classes(self) -> None:
# init the task
task = Task.init()
# Connect the config to the task
# type(self.config) -> jsonargparse.Namespace
cfg_dict = self.config.as_dict()
cfg_dict = task.connect(cfg_dict)
cfg_dict = cfg_dict._to_dict()
self.config = jsonargparse.Namespace(**cfg_dict)
if name == "main":
MyCLI(...) One could also inherit from the SaveConfigCallback to have it upload the config.yaml to ClearML 🙂 Then, when hp-optimizing the model with
HyperParameterOptimizer you can use
ParameterSet([{"General/encoder_layers": x, "General/decoder_layers.0": x} for x in range(64, 512, 64)]) to work around the variable interpolation in the config file (
decoder_layers.0 = ${encoder_layers} ). This would be equivalent to
UniformIntegerParameterRange("encoder_layers", min_value=64, max_value=512, step_size=64) ` .
For Hydra, things are easier, but I had to use http://OmegaConf.to _container
instead of http://OmegaConf.to _object
to preserve the variable interpolation strings:
` @hydra(...)
def main(config: DictConfig) -> None:
# Init the task
task = Task.init(...)
# Connect the config to ClearML
out_config = OmegaConf.to_container(config)
out_config = task.connect(out_config, "HPO")
out_config = out_config._to_dict()
config= OmegaConf.create(out_config)
# Upload the config file to ClearML
task.upload_artifact(
"config file", HydraConfig.get().run.dir + "/.hydra/config.yaml"
) `
Oh, sorry, I wrongly understand your issue. 😞
But it is the interesting one!
What comes to my mind, that https://clear.ml/docs/latest/docs/references/sdk/hpo_parameters_parameterset#class-automationparameterset can be required when there is a link between two variables. But I have never tested it and I am not ClearML developer, so do not take this advice too seriously. 🙂 Hoperfully, someone more ClearML experienced will respond you.
Hi,
I encountered similar problem. The solution was quite difficult to find, but finally we managed to update our HPO section with hyperparameters like this: https://clearml.slack.com/archives/CTK20V944/p1641372134329200?thread_ts=1640010570.080900&cid=CTK20V944
Hope this helps.
Hi CurvedHedgehog15 , thanks for replying!
I guess that one could modify the config with variable interpolation (similar to how it's done in YAML, e.g. ${encoder.layers}
) - however, it seems to be quite invasive to specify that in our trainer script 😞
I don't have issues with setting the hyperparameters - I just would like to link changes to one hyperparameter (eg. encoder.layers
) to another parameter (e.g. http://decoder.in _layers
) when optimizing over encoder.layer
Hi CostlyOstrich36
What I'm seeing is expected behavior:
In my toy example, I have a VAE which is defined by a YAML config file and parsed with PytorchLightning CLI. Part of the config defines the latent dimension (n_latents) and the number of input channels of the decoder (in_channels). These two values needs to be the same. When I just use the Lightning CLI, I can use variable interpolation with OmegaConf like this:class_path: mymodel.VAE init_args: {...} bottleneck: class_path: mymodel.Bottleneck init_args: in_channels: ${init_args.encoder.init_args.out_channels} n_latents: 256 decoder: class_path: mymodel.Decoder init_args: in_channels: ${init_args.bottleneck.init_args.n_latents} {...}
The trouble is that the variables are already inserted when ClearML updates the associated Task for training the VAE.
In the base-task for my optimization I then see this in the UI (Configuration/Hyper parameters):Args/fit.model.init_args.bottleneck.init_args.n_latents: 256
Args/fit.model.init_args.decoder.
http://init_args.in _channels: 256
which is as expected.
When I then setup a hyperparameter optimization job and I would like to modify n_latents
of my bottleneck, the number of input channels of the decoder has to be changed to the same values that was sampled for n_latents and thats my issue 🙂
Was that more clear (albeit longer)?
Edit: I have played around with a LinkedParameter, which held both a main name and linked_arg and was subclassed from clearml.automation.parameters.Parameter, but the parameters seem to be simple placeholders for the optimizer classes (e.g. in _convert_hyper_parameters_to_optuna
in clearml.automation.optuna
)
Hi GiganticMole91 ,
Can you please elaborate on this part?I can easily make ClearML treat encoder_layers as a hyperparameter, but the variables are no longer linked when they hit ClearML. I would then see Args/model.encoder_layers: 12 and Args/model.decoder_layers: 12. Is there any way to link hyperparameters in the HyperParameterOptimizer ?
What are you seeing in the UI and what were you expecting to see?