multirun is not working as expected
when I run python run.py -m env=gpu clearml.task_name=demo_all_models "model=glob(*)"
it should run remotely one run per model
this is the output I see locally
╰─ python run.py -m env=gpu clearml.task_name=demo_all_models "model=glob(*)" 2022/09/13 20:49:31 WARNING mlflow.utils.autologging_utils: You are using an unsupported version of pytorch. If you encounter errors during autologging, try upgrading / downgrading pytorch to a supported version, or try upgrading MLflow. /Users/juan/mindfoundry/git_projects/cvae/run.py:38: UserWarning: The version_base parameter is not specified. Please specify a compatability version level, or None. Will assume defaults for version 1.1 @hydra.main(config_path="configs", config_name="ou_cvae") [2022-09-13 20:49:31,808][HYDRA] Launching 3 jobs locally [2022-09-13 20:49:31,808][HYDRA] #0 : env=gpu clearml.task_name=demo_all_models model=oubetavae /Users/juan/opt/miniconda3/envs/cvae/lib/python3.9/site-packages/clearml/binding/hydra_bind.py:134: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default. See for more information. result = PatchHydra._original_run_job(*args, **kwargs) ClearML Task: created new task id=873b8743fa5e4fc381987ba6bf61e796 ClearML results page: 2022-09-13 20:49:42,169 - clearml - WARNING - Switching to remote execution, output log page [2022-09-13 20:49:43,676][HYDRA] #1 : env=gpu clearml.task_name=demo_all_models model=oucvae /Users/juan/opt/miniconda3/envs/cvae/lib/python3.9/site-packages/clearml/binding/hydra_bind.py:134: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default. See for more information. result = PatchHydra._original_run_job(*args, **kwargs) ClearML Task: created new task id=4610c1767da1404e91d73cb8f9decb47 ClearML results page: 2022-09-13 20:49:50,461 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis 2022-09-13 20:49:50,838 - clearml.Task - INFO - Finished repository detection and package analysis ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring 2022-09-13 20:49:52,706 - clearml - WARNING - Switching to remote execution, output log page [2022-09-13 20:49:54,234][HYDRA] #2 : env=gpu clearml.task_name=demo_all_models model=oulogcoshvae /Users/juan/opt/miniconda3/envs/cvae/lib/python3.9/site-packages/clearml/binding/hydra_bind.py:134: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default. See for more information. result = PatchHydra._original_run_job(*args, **kwargs) ClearML Task: created new task id=4dd7c0fda0d94636a8cdd5338c349c53 ClearML results page: 2022-09-13 20:50:01,055 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis 2022-09-13 20:50:01,419 - clearml.Task - INFO - Finished repository detection and package analysis ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring 2022-09-13 20:50:03,295 - clearml - WARNING - Switching to remote execution, output log pagebut all of those remote jobs are of the same initial model.