it would be nice if Task.connect_configuration could support custom yaml file readers for me
Yea, the config is not appearing in the webUI anymore with this method 😞
ProxyDictPostWrite._to_dict()
will recursively convert to dict and OmegaConf will not complain then
with open(path, "r") as stream: return yaml.load(stream, Loader=yaml.FullLoader)
I have a custom way of reading the config file
Doing it the other way around works:
` cfg = OmegaConf.create(read_yaml(conf_yaml_path))
config = task.connect(cfg)
type(config)
<class 'omegaconf.dictconfig.DictConfig'> `
That's because ClearML needs the hooks in that class to make sure any changes you make in that data structure are propagated back to the server when you're not in remote mode
This allows me to inject yaml files into other yaml files
Same, it also returns a ProxyDictPostWrite
, which is not supported by OmegaConf.create
This one doesn’t have
_to_dict
unfortunately
Well, I guess we can add it...
This one doesn’t have _to_dict
unfortunately
Hey SuccessfulKoala55 , unfortunately this doesn’t work, because the dict contains others dicts, and only the first level dict becomes a dict, the inner dicts still are ProxyDictPostWrite
and will make OmegaConf.create fail
I ended up dropping omegaconf altogether
I see the issue. SuccessfulKoala55 , what do you think?
So I need to have this merging of small configuration files to build the bigger one
Otherwise I can try loading the file with custom loader, save as temp file, pass the temp file to connect_configuration, it will return me another temp file with overwritten config, and then pass this new file to OmegaConf
and then call task.connect_configuration probably
From the looks of this example this should be connected automatically actually
https://github.com/allegroai/clearml/blob/master/examples/frameworks/hydra/hydra_example.py
Can you try with Task.connect()
?
https://clear.ml/docs/latest/docs/references/sdk/task#connect
Ok, so what worked for me in the end was:config = task.connect_configuration(read_yaml(conf_path)) cfg = OmegaConf.create(config._to_dict())
Regarding connect_configuration()
, reading into the docs I see that this method needs to be called before reading the config file
https://clear.ml/docs/latest/docs/references/sdk/task#connect_configuration
but then why do I have to do task.connect_configuration(read_yaml(conf_path))._to_dict()
?
Why not task.connect_configuration(read_yaml(conf_path))
simply?
I mean what is the benefit of returning ProxyDictPostWrite
instead of a dict?
But I am not sure it will connect the parameters properly, I will check now
erf, I have the same problem with ProxyDictPreWrite 😄 What is the use case of this one ?
Hey JitteryCoyote63 , the ProxyDictPostWrite
object is derived from dict
- can't you just do config = OmegaConf.create(dict(task.connect_configuration(config_dict)))
?
Yes that’s what I did initially, but eventually I decided that it’s too much complexity added for nothing really, I’d rather drop omegaconf and if one day clearml supports it out of the box take advantage of it
I am not using hydra, I am reading the conf with:config_dict = read_yaml(conf_yaml_path) config = OmegaConf.create(task.connect_configuration(config_dict))