maybe db somehow got corrupted ot smth like this? I'm clueless

https://allegro.ai/clearml/docs/docs/examples/frameworks/pytorch/notebooks/table/tabular_training_pipeline.html

yeah, I am aware of trains-agent, we are planning to start using it soon, but still, copying original training command would be useful

this definitely would be a nice addition. number of hyperparameters in our models often goes up to 100

not necessarily, command usually stays the same irrespective of the machine

copy-pasting entire training command into command line 😃

SuccessfulKoala55 grrrrr it keeps happening, I have no idea what's wrong

same here, no live logs

give me a sec

problem is solved. I had to replace /opt/trains/data/fileserver to /opt/clearml/data/fileserver in Agent configuration, and replace trains to clearml in Requirements

thanks!

two more questions about cleanup if you don't mind:
what if for some old tasks I get WARNING:root:Could not delete Task ID=a0908784a2a942c3812f947ec1f32c9f, 'Task' object has no attribute 'delete'? What's the best way of cleaning them? What is the recommended way of providing S3 credentials to cleanup task?

what if cleanup service is launched using ClearML-Agent Services container (part of the ClearML server)? adding clearml.conf to the home directory doesn't help

oh wow, I didn't see delete_artifacts_and_models option

I guess we'll have to manually find old artifacts that are related to already deleted tasks

we already have cleanup service set up and running, so we should be good from now on

btw, are there any examples of exporting metrics using Python client? I could only find last_metrics attribute of the task

some of the POST requests "tasks.get_all_ex" fail as far as I can see

thanks!

tail of the api server log

running
docker network prunebefore starting the containers kind of helped. I still see an error when I'm comparing > 20 experiments, but at least trains works okay after that, and there are no connection pool limit errors in the logs

Error
Failed to get Scalar Charts

0.16.1

nope, the only changes to config that we made are adding web-auth and non-responsive tasks watchdog

just in case, this warning disappeared after I https://stackoverflow.com/questions/49638699/docker-compose-restart-connection-pool-full

btw, there are "[2020-09-02 15:15:40,331] [9] [WARNING] [urllib3.connectionpool] Connection pool is full, discarding connection: elasticsearch" in the apiserver logs again

m5.xlarge EC2 instance (4 vCPUs, 16 GB RAM), 100GB disk

any suggestions on how to fix it?

network logs

yes. we upload artifacts to Yandex.Cloud S3 using ClearML. we set " s3://storage.yandexcloud.net/clearml-models " as output uri parameter and add this section to the config:
{
host: " http://storage.yandexcloud.net "
key: "KEY"
secret:"SECRET_KEY",
secure: true
}

this works like a charm. but download button in UI is not working

we do log a lot of the different metrics, maybe this can be part of the problem

after the very first click, there is a popup with credentials request. nothing happens after that