maybe db somehow got corrupted ot smth like this? I'm clueless

1.3.0

thanks!

nope, the only changes to config that we made are adding web-auth and non-responsive tasks watchdog

just in case, this warning disappeared after I https://stackoverflow.com/questions/49638699/docker-compose-restart-connection-pool-full

hoooraaaay

that's right, I have 4 GPUs and 4 workers. but what if I want to run two jobs simultaneously at the same GPU

yes, this is the use case, I think we can use smth like Redis for this communication

btw, there are "[2020-09-02 15:15:40,331] [9] [WARNING] [urllib3.connectionpool] Connection pool is full, discarding connection: elasticsearch" in the apiserver logs again

well, the server wouldn't work without them?

this is how it looks if I zoom in on the epochs that ran before the crash

overwriting this value is not ideal though, because for :monitor:gpu and :monitor:machine values I would like to continue from the latest iteration

but for the metrics, I explicitly pass the number of epoch that my training is currently on. it'ls kind of weird that it adds offset to the values that are explicitly reported, no?

I use Docker for training, which means that log_dir contents are removed for the continued experiment btw

[2020-06-09 16:03:19,851] [8] [ERROR] [trains.mongo.initialize] Failed creating fixed user John Doe: 'key'

yes

we often do ablation studies with more than 50 experiments, and it was very convenient to compare their dynamics at the different epochs

yeah, we did. let me check if explicitly setting credentials helps

hmmm allegroai/trains:latest whatever it is

SuccessfulKoala55 grrrrr it keeps happening, I have no idea what's wrong

WARNING: You are using pip version 20.1.1; however, version 20.3.3 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

trains_agent: ERROR: Connection Error: it seems *api_server* is misconfigured. Is this the TRAINS API server http://apiserver:8008 ?

http://OUR_IP:8081 http://OUR_IP:8080 http://apiserver:8008
WARNING: You are using pip version 20.1.1; however, version 20.3.3 is available.
`...

oh wow, I didn't see delete_artifacts_and_models option

I guess we'll have to manually find old artifacts that are related to already deleted tasks

sorry that I keep bothering you, I love ClearML and try to promote it whenever I can, but this thing is a real pain in the ass 😃

sounds like an overkill for this problem, but I don’t see any other pretty solution 😃

magic

dnk if it's relevant, but I also added a new user to apiserver.conf today

yeah, that sounds right! thanks, will try

wow, thanks, just updated our server!
can't seem to find these metrics snapshot plots =) how do I plot one?

yes

thanks! we copy S3 URLs quite often. I know that it’s better to avoid double spaces in task names, but shit happens 😄

nope, didn't work =(

docker mode
different ids