@<1523701070390366208:profile|CostlyOstrich36> Hi
I tried this example but not seems im doing it right ....
How does this two scripts communicating between themselves ?
base_template_keras_simple.py
hyper_parameter_optimizer.py
im not understand which is should run first ?
Hi @<1734744942921650176:profile|RipeBee29> , you can find an example here:
None
I suggest reviewing what other examples are there
So created queue ... now im getting
python scripts/hyper_parameter_optimizer.py
ClearML Task: created new task id=2ba36b52e6004514a41dc52bb2ba169c
ClearML results page:
[I 2024-08-18 13:31:41,133] A new study created in memory with name: 2ba36b52e6004514a41dc52bb2ba169c
Progress report #0 completed, sleeping for 0.25 minutes
2024-08-18 13:31:41,370 - clearml.automation.optimization - INFO - Creating new Task: {'General/layer_1': 256, 'General/layer_2': 384, 'General/batch_size': 160, 'General/epochs': 30}
2024-08-18 13:31:41,627 - clearml.automation.optimization - INFO - Creating new Task: {'General/layer_1': 512, 'General/layer_2': 384, 'General/batch_size': 128, 'General/epochs': 30}
Progress report #1 completed, sleeping for 0.2 minutes
Progress report #2 completed, sleeping for 0.2 minutes
Progress report #3 completed, sleeping for 0.2 minutes
Progress report #4 completed, sleeping for 0.2 minutes
Progress report #5 completed, sleeping for 0.2 minutes
Progress report #6 completed, sleeping for 0.2 minutes
Progress report #7 completed, sleeping for 0.2 minutes
Progress report #8 completed, sleeping for 0.2 minutes
Progress report #9 completed, sleeping for 0.2 minutes
Progress report #10 completed, sleeping for 0.2 minutes
Progress report #11 completed, sleeping for 0.2 minutes
Progress report #12 completed, sleeping for 0.2 minutes
[W 2024-08-18 13:34:14,978] Trial 0 failed with parameters: {'General/layer_1': 256, 'General/layer_2': 384, 'General/batch_size': 160, 'General/epochs': 30} because of the following error: TypeError("'NoneType' object is not subscriptable").
Traceback (most recent call last):
File "/home/razoffice/corsound_vfm/venv_raz/lib/python3.10/site-packages/optuna/study/_optimize.py", line 196, in _run_trial
value_or_values = func(trial)
File "/home/razoffice/corsound_vfm/venv_raz/lib/python3.10/site-packages/clearml/automation/optuna/optuna.py", line 92, in objective
objective_metric = objective_metric[0]
TypeError: 'NoneType' object is not subscriptable
[W 2024-08-18 13:34:14,980] Trial 0 failed with value None.
Progress report #13 completed, sleeping for 0.2 minutes
Progress report #14 completed, sleeping for 0.2 minutes
Progress report #15 completed, sleeping for 0.2 minutes
[W 2024-08-18 13:34:56,087] Trial 1 failed with parameters: {'General/layer_1': 512, 'General/layer_2': 384, 'General/batch_size': 128, 'General/epochs': 30} because of the following error: TypeError("'NoneType' object is not subscriptable").
Traceback (most recent call last):
File "/home/razoffice/corsound_vfm/venv_raz/lib/python3.10/site-packages/optuna/study/_optimize.py", line 196, in _run_trial
value_or_values = func(trial)
File "/home/razoffice/corsound_vfm/venv_raz/lib/python3.10/site-packages/clearml/automation/optuna/optuna.py", line 92, in objective
objective_metric = objective_metric[0]
TypeError: 'NoneType' object is not subscriptable
[W 2024-08-18 13:34:56,088] Trial 1 failed with value None.
Exception in thread Thread-5 (_daemon):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
....
File "/home/razoffice/corsound_vfm/venv_raz/lib/python3.10/site-packages/optuna/study/_optimize.py", line 247, in _run_trial
raise func_err
File "/home/razoffice/corsound_vfm/venv_raz/lib/python3.10/site-packages/optuna/study/_optimize.py", line 196, in _run_trial
value_or_values = func(trial)
File "/home/razoffice/corsound_vfm/venv_raz/lib/python3.10/site-packages/clearml/automation/optuna/optuna.py", line 92, in objective
objective_metric = objective_metric[0]
TypeError: 'NoneType' object is not subscriptable
['333ed1cf667c4ba390844b757a2625ef', '04921bcae42c403f99de46f1c584372d']
`Study.stop` is supposed to be invoked inside an objective function or a callback.
We are done, good bye
I didnt modified your scripts , only the execution_queue variable
Sure i did it but im getting the following
A new study created in memory with name: 8ea132f808934f1a962aeb89cff31bd0
Progress report #0 completed, sleeping for 0.25 minutes
2024-08-18 12:55:24,959 - clearml.automation.optimization - INFO - Creating new Task: {'General/layer_1': 128, 'General/layer_2': 256, 'General/batch_size': 160, 'General/epochs': 30}
2024-08-18 12:55:25,030 - clearml.automation.job - WARNING - Error enqueuing Task <clearml.task.Task object at 0x723c45320610> to 1xGPU: Could not find queue named "1xGPU"
[W 2024-08-18 12:55:25,030] Trial 0 failed with parameters: {'General/layer_1': 128, 'General/layer_2': 256, 'General/batch_size': 160, 'General/epochs': 30} because of the following error: The value None could not be cast to float..
[W 2024-08-18 12:55:25,031] Trial 0 failed with value None.
2024-08-18 12:55:25,287 - clearml.automation.optimization - INFO - Creating new Task: {'General/layer_1': 384, 'General/layer_2': 128, 'General/batch_size': 96, 'General/epochs': 30} ...
[W 2024-08-18 12:55:26,702] Trial 9 failed with value None.
['0effcb51e242499f9abb0f7c8d6eac60', '067dd618ef4c4db4adeb96553d7e1d44', '809ed889e43c4720b5d3d78883b69fea']
We are done, good bye
I suggest you review ClearML onboarding videos on the Youtube channel and read about the basics of the agent/orchestration.
You also need an agent listening to the queue you're enqueuing to
You need to run the base first and then the optimizer. The optimizer will clone the base and enqueue it. Read further here - None
Please see the error:
2024-08-18 12:55:25,030 - clearml.automation.job - WARNING - Error enqueuing Task <clearml.task.Task object at 0x723c45320610> to 1xGPU: Could not find queue named "1xGPU"
You don't have a queue called 1xGPU.