Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Executed A Hyperparameter Optimization Task, But Just One Single Experiment Was Performed And Nothing. The Task Wasn'T Terminated Automatically, But Still Not Starting The Next Experiments. What'S Wrong Here? How Can I Fix This?

I executed a hyperparameter optimization task, but just one single experiment was performed and nothing. The task wasn't terminated automatically, but still not starting the next experiments.
What's wrong here? How can I fix this?

  
  
Posted 5 months ago
Votes Newest

Answers 10


fyi,
I set the options for HyperParameterOptimizer() like,

  • compute_time_limit=None,
  • total_max_jobs=100,
  • min_iteration_per_job=NOne,
  • max_iteration_per_job=NOne,
  • max_number_of_concurrent_tasks=1
  
  
Posted 5 months ago

plus, the first experiment terminated with early stopping.

  
  
Posted 5 months ago

Was "task.close()" called for the early-stopped task?
What is the experiment status in Web UI?

  
  
Posted 5 months ago

DangerousStarfish38 , can you provide logs please?

  
  
Posted 5 months ago

CostlyOstrich36 here it is!

  
  
Posted 5 months ago

TroubledCamel37 No, I didn't add "task.close()" in the code. This link is what I followed.

Even after completing one experiment, the console and UI don't seem to terminate the task.

  
  
Posted 5 months ago

TroubledCamel37 but, I guess task.close() would terminate the optimization task, not the single experiment. am I misunderstanding something? 😭

  
  
Posted 5 months ago

DangerousStarfish38 Yep, you are right, according to the docs, the optimizer.stop() should be used, not task.close(). Sorry for confusing.
I guess the issue is in connectivity/auth problems between ClearML components - there are many timeout messages in the log. I have similar messages for fileserver container, not yet resolved.

  
  
Posted 5 months ago

TroubledCamel37 Thanks! I'll look over the connectivity issue that you said.

  
  
Posted 5 months ago

Yeah, the problem was about fileserver connection like you said!
I was running the experiment in remote server, and solved the issue by opening the port for fileserver! Thanks!

  
  
Posted 5 months ago
469 Views
10 Answers
5 months ago
5 months ago
Tags