I Am Trying To Do Some Hyperparameter Optimization Using A Script And Have Closely Followed The Example On Github. I Keep Getting This Warning When Using Optunaoptimizer:

Answered

I am trying to do some hyperparameter optimization using a script and have closely followed the example on Github. I keep getting this warning when using OptunaOptimizer:

`Study.stop` is supposed to be invoked inside an objective function or a callback.

which means that the optimization runs all have the status pending rather than completed . I have this at the end of my script:

  optimizer.start()
  optimizer.wait()
  top_exp = optimizer.get_top_experiments(top_k=3)
  print('Top experiments are:')
  print([t.id for t in top_exp])
  optimizer.stop()

so I'm not sure why I'm getting the warning.

I am using ClearML 1.12.0
Would anyone have suggestions on how I can solve this?

  				
Posted 
	one month ago

					More
				  		
  Report
		
					IcyFlamingo4
				
					0
					 × 1

Votes Newest

Answers 3

Ok will try that, thank you

  				
Posted 
	one month ago

					More
				  		
  Report
		
					IcyFlamingo4
				
					0
					 × 1

Hi @<1523701087100473344:profile|SuccessfulKoala55> I've updated to clearml==1.16.2 and I'm still getting the same warnings. I'm now getting an additional error which may be linked:

Exception in thread Thread-6 (_report_daemon):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/automation/optimization.py", line 2005, in _report_daemon
    self.stop(wait_for_reporter=False)
  File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/automation/optimization.py", line 1578, in stop
    j.abort()
  File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/automation/job.py", line 104, in abort
    Task.dequeue(self.task)
  File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/task.py", line 1518, in dequeue
    res = cls._send(session=session, req=req)
  File "/home/anirudh/ml_experiments/.venv/lib/python3.10/site-packages/clearml/backend_interface/base.py", line 107, in _send
    raise SendError(res, error_msg)
clearml.backend_interface.session.SendError: Action failed <400/121: tasks.dequeue/v1.0 (Failed changing task status. probably someone changed it before you: task_id=5da187b920f14c8cbb8183639f8b96fc, current_status=queued, new_status=created)> (task=5da187b920f14c8cbb8183639f8b96fc)

That task ID that is referenced at the end of the trace is one of the draft tasks started by the optimizer.

Do you have any suggestions?
Thanks

  				
Posted 
	one month ago

					More
				  		
  Report
		
					IcyFlamingo4
				
					0
					 × 1

Hi @<1727859568827961344:profile|IcyFlamingo4> , I would recommend to first upgrade ClearML SDK to the latest version (currently v1.16.2) and see if it helps

  				
Posted 
	one month ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Write your answer

139 Views

3 Answers

one month ago