Unanswered
I Am Trying To Do Some Hyperparameter Optimization Using A Script And Have Closely Followed The Example On Github. I Keep Getting This Warning When Using Optunaoptimizer:
Hi @<1523701087100473344:profile|SuccessfulKoala55> I've updated to clearml==1.16.2
and I'm still getting the same warnings. I'm now getting an additional error which may be linked:
Exception in thread Thread-6 (_report_daemon):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/automation/optimization.py", line 2005, in _report_daemon
self.stop(wait_for_reporter=False)
File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/automation/optimization.py", line 1578, in stop
j.abort()
File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/automation/job.py", line 104, in abort
Task.dequeue(self.task)
File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/task.py", line 1518, in dequeue
res = cls._send(session=session, req=req)
File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/backend_interface/base.py", line 107, in _send
raise SendError(res, error_msg)
clearml.backend_interface.session.SendError: Action failed <400/121: tasks.dequeue/v1.0 (Failed changing task status. probably someone changed it before you: task_id=5da187b920f14c8cbb8183639f8b96fc, current_status=queued, new_status=created)> (task=5da187b920f14c8cbb8183639f8b96fc)
That task ID that is referenced at the end of the trace is one of the draft tasks started by the optimizer.
Do you have any suggestions?
Thanks
36 Views
0
Answers
3 months ago
3 months ago