I change the arguments in Web UI, but it looks like they are not parsed by trains

I use the following Base Docker image setting: my-docker-hub/my-repo:latest -v /data/project/data:/data

the code that is used for training the model is also inside the image

example of the failed experiment

nope, same problem even after creating a new experiment from scratch

thank you, I'll let you know if setting it to zero worked

python3 slack_alerts.py --channel trains-alerts --slack_api "OUR_KEY" --include_completed_experiments --include_manual_experiments

LOL
wow 😃
I was trying to find how to create a queue using CLI 😃

docker mode
different ids

same here, changing arguments in the Args section of Hyperparameters doesn’t work, training script starts with the default values.

trains 0.16.0
trains-agent 0.16.0
trains-server 0.16.0

I updated the version in the Installed packages section before starting the experiment

it prints an empty dict

I’m doing Task.init() in the script, maybe it somehow resets connected parameters… but it used to work before, weird

this definitely would be a nice addition. number of hyperparameters in our models often goes up to 100

I guess, this could overcomplicate ui, I don't see a good solution yet.

as a quick hack, we can just use separate name (eg "best_val_roc_auc") for all metric values for the current best checkpoint. then we can just add columns with the last value of this metric

thank you 😃

nope, old clenup task fails with trains_agent: ERROR: Could not find task id=e7725856e9a04271aab846d77d6f7d66 (for host: )
Exception: 'Tasks' object has no attribute 'id

weirdly enough, curl http://apiserver:8008 from inside the container works

exactly

all our workers went down after starting the slack bot, is it expected?)

running
docker network prunebefore starting the containers kind of helped. I still see an error when I'm comparing > 20 experiments, but at least trains works okay after that, and there are no connection pool limit errors in the logs

btw, are there any examples of exporting metrics using Python client? I could only find last_metrics attribute of the task

I don’t connect anything explicitly, I’m using argparse, it used to work before the update

btw, there are "[2020-09-02 15:15:40,331] [9] [WARNING] [urllib3.connectionpool] Connection pool is full, discarding connection: elasticsearch" in the apiserver logs again

I updated S3 credentials, I'll check if they work later

it doesn't explain inability to delete logged images and texts though

we do log a lot of the different metrics, maybe this can be part of the problem

if you click on the experiment name here, you get 404 because link looks like this:
https://DOMAIN/projects/PROJECT_ID/EXPERIMENT_ID
when it should look like this:
https://DOMAIN/projects/PROJECT_ID/experiments/EXPERIMENT_ID

{
username: "username"
password: "password"
name: "John Doe"
},

got it, thanks!

not necessarily, there are rare cases when container keeps running after experiment is stopped or aborted

will do!

this is how the interface looks

just DMed you a screenshot where you can see a part of the token