I Am Trying To Do Some Hyperparameter Optimization Using A Script And Have Closely Followed The Example On Github. I Keep Getting This Warning When Using Optunaoptimizer:

Answered

I am trying to do some hyperparameter optimization using a script and have closely followed the example on Github. I keep getting this warning when using OptunaOptimizer:

`Study.stop` is supposed to be invoked inside an objective function or a callback.

which means that the optimization runs all have the status pending rather than completed . I have this at the end of my script:

  optimizer.start()
  optimizer.wait()
  top_exp = optimizer.get_top_experiments(top_k=3)
  print('Top experiments are:')
  print([t.id for t in top_exp])
  optimizer.stop()

so I'm not sure why I'm getting the warning.

I am using ClearML 1.12.0
Would anyone have suggestions on how I can solve this?

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					IcyFlamingo4
				
					0
					 × 1

Votes Newest

Answers 3

Hi @<1523701087100473344:profile|SuccessfulKoala55> I've updated to clearml==1.16.2 and I'm still getting the same warnings. I'm now getting an additional error which may be linked:

Exception in thread Thread-6 (_report_daemon):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/automation/optimization.py", line 2005, in _report_daemon
    self.stop(wait_for_reporter=False)
  File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/automation/optimization.py", line 1578, in stop
    j.abort()
  File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/automation/job.py", line 104, in abort
    Task.dequeue(self.task)
  File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/task.py", line 1518, in dequeue
    res = cls._send(session=session, req=req)
  File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/backend_interface/base.py", line 107, in _send
    raise SendError(res, error_msg)
clearml.backend_interface.session.SendError: Action failed <400/121: tasks.dequeue/v1.0 (Failed changing task status. probably someone changed it before you: task_id=5da187b920f14c8cbb8183639f8b96fc, current_status=queued, new_status=created)> (task=5da187b920f14c8cbb8183639f8b96fc)

That task ID that is referenced at the end of the trace is one of the draft tasks started by the optimizer.

Do you have any suggestions?
Thanks

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					IcyFlamingo4
				
					0
					 × 1

Hi @<1727859568827961344:profile|IcyFlamingo4> , I would recommend to first upgrade ClearML SDK to the latest version (currently v1.16.2) and see if it helps

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Ok will try that, thank you

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					IcyFlamingo4
				
					0
					 × 1

Write your answer

317 Views

3 Answers

4 months ago