python3 slack_alerts.py --channel trains-alerts --slack_api "OUR_KEY" --include_completed_experiments --include_manual_experiments

new icons are slick, it would be even better if you could upload custom icons for the different projects

we are working on the medical projects, so probably images of the different body parts 😃

yeah, I am aware of trains-agent, we are planning to start using it soon, but still, copying original training command would be useful

not necessarily, command usually stays the same irrespective of the machine

copy-pasting entire training command into command line 😃

this definitely would be a nice addition. number of hyperparameters in our models often goes up to 100

AnxiousSeal95 yeah, got it! thanks!

btw, there are "[2020-09-02 15:15:40,331] [9] [WARNING] [urllib3.connectionpool] Connection pool is full, discarding connection: elasticsearch" in the apiserver logs again

network logs

tail of the api server log

btw, are there any examples of exporting metrics using Python client? I could only find last_metrics attribute of the task

any suggestions on how to fix it?

we do log a lot of the different metrics, maybe this can be part of the problem

running
docker network prunebefore starting the containers kind of helped. I still see an error when I'm comparing > 20 experiments, but at least trains works okay after that, and there are no connection pool limit errors in the logs

okay, give me a sec

Error
Failed to get Scalar Charts

0.16.1

thanks!

some of the POST requests "tasks.get_all_ex" fail as far as I can see

1 - yes, of course =) but it would be awesome if you could customize the content - to include key metrics and hyperparameters, for example
3 - hooooooraaaay

yeah, we've used pipelines in other scenarios. might be a good fit here. thanks!

m5.xlarge EC2 instance (4 vCPUs, 16 GB RAM), 100GB disk

I’ll let you know 😃

JIC - trains still works after that, it's just that the new user is not added and hence is not able to login

0.15.0

[2020-06-09 16:03:19,851] [8] [ERROR] [trains.mongo.initialize] Failed creating fixed user John Doe: 'key'

maybe db somehow got corrupted ot smth like this? I'm clueless

{
username: "username"
password: "password"
name: "John Doe"
},

kind of