we often do ablation studies with more than 50 experiments, and it was very convenient to compare their dynamics at the different epochs

I use the following Base Docker image setting: my-docker-hub/my-repo:latest -v /data/project/data:/data

the code that is used for training the model is also inside the image

I decided to restart the containers one more time, this is what I got.

I had to restart Docker service to remove the containers

JIC - trains still works after that, it's just that the new user is not added and hence is not able to login

example of the failed experiment

it might be that there is not enough space on our SSD, experiments cache a lot of preprocessed data during the first epoch...

0.15.0

nope, same problem even after creating a new experiment from scratch

thank you, I'll let you know if setting it to zero worked

I'll get back to you with the logs when the problem occurs again

as a sidenote, I am not able to pull the newest release, looks like it's not pushed?
"Error response from daemon: manifest for allegroai/trains:0.14.2 not found"

1 - yes, of course =) but it would be awesome if you could customize the content - to include key metrics and hyperparameters, for example
3 - hooooooraaaay

we do log a lot of the different metrics, maybe this can be part of the problem

does this mean that setting initial iteration to 0 should help?

yeah, I think this is what I need!

it also happens sometimes during the run when tensorboard is trying to write smth to the disk and there are multiple experiments running. so it must be smth similar to the scenario you're describing, but I have no idea how it can happen since I'm running four separate workers

not necessarily, command usually stays the same irrespective of the machine

I've already pulled new images from trains-server, let's see if the initial issue occurs again. thank for the fast response guys!

thanks! I need to read all parts of documentation really carefully =) for some reason, couldn't find this section

our GPUs are 48GB, so it's quite wasteful to only run one job per GPU
yeah, I'm aware of that, I would have to make sure they don't fail to infamous CUDA out of memory, but still

is it in documentation somewhere?

all our workers went down after starting the slack bot, is it expected?)

I assume, temporary fix is to switch to trains-server?

I don’t connect anything explicitly, I’m using argparse, it used to work before the update

https://cloud.yandex.com/en-ru/docs/storage/s3/

we are working on the medical projects, so probably images of the different body parts 😃

dnk if it's relevant, but I also added a new user to apiserver.conf today

thanks! this bug and cloning problem seem to be fixed

yeah, I am aware of trains-agent, we are planning to start using it soon, but still, copying original training command would be useful