yeah

These are the last_metrics values the task object has

Hi PungentLouse55
it depends on the trains-server version you are running.
If the trains-server >= 0.16 then you have to add "Args/" prefix. If you are running an older version, then you should not add any prefix.

you are correct, I was referring to the template experiment

PungentLouse55 could you test again with the latest from the GitHub? I think the issue should be solved:
pip install git+

Yes 😅 It actually worked. Thank you! Now I got values from scalars

Hi PungentLouse55
Are you referring to the example code ?

How can I get them? I think I follow the example from documentation, but I cant get it.
Is it ok that my template experiment is now at draft 'state'?

I'll make sure we fix the example, because as you pointed, it is broken :(

A few epochs is just fine

PungentLouse55 I'm checking something here, you might stumbled on a bug in parameter overriding. Updating here soon ...

Yes, sorry, that wasn't clear 🙂

PungentLouse55 you can find the metrics in the "original" (aka base template) experiment.

In order for the sample to work you have to run the template experiment once. Then the HP optimizer will find the best HP for it.

`

Example use case:

an_optimizer = HyperParameterOptimizer(
# This is the experiment we want to optimize
base_task_id=args['template_task_id'],
# here we define the hyper-parameters to optimize
hyper_parameters=[
UniformIntegerParameterRange('General/layer_1', min_value=128, max_value=512, step_size=128),
UniformIntegerParameterRange('General/layer_2', min_value=128, max_value=512, step_size=128),
DiscreteParameterRange('General/batch_size', values=[96, 128, 160]),
DiscreteParameterRange('General/epochs', values=[30]),
],
# this is the objective metric we want to maximize/minimize
objective_metric_title='epoch_accuracy',
objective_metric_series='epoch_accuracy',
# now we decide if we want to maximize it or minimize it (accuracy we maximize)
objective_metric_sign='max',
# let us limit the number of concurrent experiments,
# this in turn will make sure we do dont bombard the scheduler with experiments.
# if we have an auto-scaler connected, this, by proxy, will limit the number of machine
max_number_of_concurrent_tasks=2,
# this is the optimizer class (actually doing the optimization)
# Currently, we can choose from GridSearch, RandomSearch or OptimizerBOHB (Bayesian optimization Hyper-Band)
# more are coming soon...
optimizer_class=aSearchStrategy,
# Select an execution queue to schedule the experiments for execution
execution_queue='1xGPU',
# Optional: Limit the execution time of a single experiment, in minutes.
# (this is optional, and if using OptimizerBOHB, it is ignored)
time_limit_per_job=10.,
# Check the experiments every 12 seconds is way too often, we should probably set it to 5 min,
# assuming a single experiment is usually hours...
pool_period_min=0.2,
# set the maximum number of jobs to launch for the optimization, default (None) unlimited
# If OptimizerBOHB is used, it defined the maximum budget in terms of full jobs
# basically the cumulative number of iterations will not exceed total_max_jobs * max_iteration_per_job
total_max_jobs=10,
# set the minimum number of iterations for an experiment, before early stopping.
# Does not apply for simple strategies such as RandomSearch or GridSearch
min_iteration_per_job=10,
# Set the maximum number of iterations for an experiment to execute
# (This is optional, unless using OptimizerBOHB where this is a must)
max_iteration_per_job=30,
) `

I'll try again, but I did it in this way 😢
Maybe I am wrong and did something wrong

This is a screen with messages of an optimization process

PungentLouse55 , make sure you fix the metric objective and args:
Add "General/" prefix to the list of arguments to optimize, and change the objective metric from "Accuracy" to "epoch_accuracy"

Hope you are not tired of me. But I am using trains 0.16.1 and adding prefix does not work. I found the place where dict with keys <prefix/key>:value are transformed to nested dict with <prefix>:{<key>: value} (see screenshots). Im sorry for my annoyance but I believe there is misandastanding between us and you think that prefixes work 🙂

Cause I ran for a few epochs only

Thank you! I'll try

I want to optimizer hyperparameters with trains.automation but: ...

Yes you are correct, in case of the example code, it should be "General/..." if you have ArgParser, it should be "Args/..." Yes it looks like the metric is wrong, it should be "epoch_accuracy" & "epoch_accuracy"

PungentLouse55 from the screenshot I assume the experiment template you are trying to optimize is not the one from the trains/examples 🙂
In that case, and based on the screenshots, the prefix is "Args/" as this is the section name.
Regrading objective metric, again based on your screenshots:
objective_metric_title="Accuracy" objective_metric_series="Validation"Make sense ?

here's my script

Unfortunately I still can't get it

Adding General prefix for parameters doesn't work as task parameters have no prefixes. The also doesn't have 'General' key returned (pictures 1, 2 are screen shots of my base experiment, picture 3 is key of returned task parameters dictionary) The last_metrics argument is still empty. But my template experiment actually has reported scalars (picture 4) and I use right experiment id (picture 5)

Could you please answer the last question? 🙃
I'm I right that the is a bug with the first situation I've described ('Args' parameter)? Or I do smth wrong and It should work with prefixes? Because it does not work if I add prefix

I'm sorry but I didn't get you about original experiment. By original you mean the experiment I use as a template?

Hi PungentLouse55

Hope you are not tired of me

Lol 🙂 No worries

I am using trains 0.16.1

Are you referring to the trains-server version or the python package ? (they are not the same and can be of totally different versions)

YEY!

Should I run the template experiment till the end (I mean when it'not not improving) or I can run for a few epochs?