LOL
wow 😃
I was trying to find how to create a queue using CLI 😃

I'm so happy to see that this problem has been finally solved!

yeah, I was thinking mainly about AWS. we use force to make sure we are using the correct latest checkpoint, but this increases costs when we are running a lot of experiments

yes

original task name contains double space -> saved checkpoint also contains double space -> MODEL URL field in model description of this checkpoint in ClearML converts double space into single space. so when you copy & paste it somewhere, it'll be incorrect

sounds like an overkill for this problem, but I don’t see any other pretty solution 😃

nice, thanks! I'll check if it solves the issue first thing tomorrow in the morning

default docker-compose

new version worked

do you have any idea why cleanup task keeps failing then (it used to work before the update)

thank you 😃

well, the server wouldn't work without them?

yeah, we did. let me check if explicitly setting credentials helps

nope, old clenup task fails with trains_agent: ERROR: Could not find task id=e7725856e9a04271aab846d77d6f7d66 (for host: )
Exception: 'Tasks' object has no attribute 'id

weirdly enough, curl http://apiserver:8008 from inside the container works

magic

yeap, it works 😍

will do

WARNING: You are using pip version 20.1.1; however, version 20.3.3 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

trains_agent: ERROR: Connection Error: it seems *api_server* is misconfigured. Is this the TRAINS API server http://apiserver:8008 ?

http://OUR_IP:8081 http://OUR_IP:8080 http://apiserver:8008
WARNING: You are using pip version 20.1.1; however, version 20.3.3 is available.
`...

awesome!

nice, thanks

wow, thanks, just updated our server!
can't seem to find these metrics snapshot plots =) how do I plot one?

AnxiousSeal95 yeah, got it! thanks!

1 - yes, of course =) but it would be awesome if you could customize the content - to include key metrics and hyperparameters, for example
3 - hooooooraaaay

we are working on the medical projects, so probably images of the different body parts 😃

JIC - trains still works after that, it's just that the new user is not added and hence is not able to login

we've already restarted everything, so I don't have any logs on hands right now. I'll let you know if we face any problems 😃 slack bot works though! 🎉

python3 slack_alerts.py --channel trains-alerts --slack_api "OUR_KEY" --include_completed_experiments --include_manual_experiments

new icons are slick, it would be even better if you could upload custom icons for the different projects

all our workers went down after starting the slack bot, is it expected?)

yes