─ python run.py -m env=gpu clearml.task_name=connect_test "model=glob(*)" trainer_params.max_epochs=5 2022/09/14 01:10:07 WARNING mlflow.utils.autologging_utils: You are using an unsupported version of pytorch. If you encounter errors during autologging, try upgrading / downgrading pytorch to a supported version, or try upgrading MLflow. /Users/juan/mindfoundry/git_projects/cvae/run.py:38: UserWarning: The version_base parameter is not specified. Please specify a compatability version level, or None. Will assume defaults for version 1.1 @hydra.main(config_path="configs", config_name="ou_cvae") [2022-09-14 01:10:07,712][HYDRA] Launching 3 jobs locally [2022-09-14 01:10:07,712][HYDRA] #0 : env=gpu clearml.task_name=connect_test model=oubetavae trainer_params.max_epochs=5 /Users/juan/opt/miniconda3/envs/cvae/lib/python3.9/site-packages/clearml/binding/hydra_bind.py:134: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default. See for more information. result = PatchHydra._original_run_job(*args, **kwargs) ClearML Task: created new task id=afd819adc5e84458bd1a271ab786da05 ClearML results page: {'params': {'in_channels': 1, 'num_classes': 64, 'latent_dim': 128, 'img_size': 128, 'loss_type': 'B', 'gamma': 10.0, 'max_capacity': 25, 'Capacity_max_iter': 10000}, 'name': 'OUBetaVAE'} ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring 2022-09-14 01:10:18,785 - clearml - WARNING - Switching to remote execution, output log page [2022-09-14 01:10:20,420][HYDRA] #1 : env=gpu clearml.task_name=connect_test model=oucvae trainer_params.max_epochs=5 /Users/juan/opt/miniconda3/envs/cvae/lib/python3.9/site-packages/clearml/binding/hydra_bind.py:134: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default. See for more information. result = PatchHydra._original_run_job(*args, **kwargs) ClearML Task: created new task id=5f07dcfa88b946c5b67f109922e7dcfe ClearML results page: {'params': {'in_channels': 1, 'num_classes': 64, 'latent_dim': 128, 'img_size': 128}, 'name': 'OUCVAE'} 2022-09-14 01:10:27,769 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring 2022-09-14 01:10:28,157 - clearml.Task - INFO - Finished repository detection and package analysis 2022-09-14 01:10:30,180 - clearml - WARNING - Switching to remote execution, output log page [2022-09-14 01:10:31,793][HYDRA] #2 : env=gpu clearml.task_name=connect_test model=oulogcoshvae trainer_params.max_epochs=5 /Users/juan/opt/miniconda3/envs/cvae/lib/python3.9/site-packages/clearml/binding/hydra_bind.py:134: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default. See for more information. result = PatchHydra._original_run_job(*args, **kwargs) ClearML Task: created new task id=40f8a8d8830f45b99e214edb237ad4c0 ClearML results page: {'params': {'in_channels': 1, 'num_classes': 64, 'latent_dim': 128, 'img_size': 128, 'alpha': 10.0, 'beta': 1.0}, 'name': 'OULogCoshVAE'} 2022-09-14 01:10:39,159 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring 2022-09-14 01:10:39,560 - clearml.Task - INFO - Finished repository detection and package analysis 2022-09-14 01:10:41,553 - clearml - WARNING - Switching to remote execution, output log pagehere are the prints. The tasks each have different models, but the remote versions all seem to start with a model at random. Two with same model, and one different
but I still have the problem if I try to run locally for debugging purposes
clearml-agent execute --id ...
Is this still an issue ? this is basically the same as the remote execution, maybe you should add the container (if the agent is running in docker mode) --docker ?
