yeah, backups take much longer, and we had to increase our EC2 instance volume size twice because of these indices

got it, thanks, will try to delete older ones

yeap, it works 😍

thanks!

yeah, server (1.0.0) and client (1.0.1)

yeah, we did. let me check if explicitly setting credentials helps

thank you

will do

SuccessfulKoala55 grrrrr it keeps happening, I have no idea what's wrong

yeah, I think this is what I need!

two more questions about cleanup if you don't mind:
what if for some old tasks I get WARNING:root:Could not delete Task ID=a0908784a2a942c3812f947ec1f32c9f, 'Task' object has no attribute 'delete'? What's the best way of cleaning them? What is the recommended way of providing S3 credentials to cleanup task?

isn't this parameter related to communication with ClearML Server? I'm trying to make sure that checkpoint will be downloaded from AWS S3 even if there are temporary connection problems

there's https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig parameter in boto3, but I'm not sure if there's an easy way to pass this parameter to StorageManager

[2020-06-09 16:03:19,851] [8] [ERROR] [trains.mongo.initialize] Failed creating fixed user John Doe: 'key'

not necessarily, command usually stays the same irrespective of the machine

we're using EC2

AnxiousSeal95 yeah, got it! thanks!

I’ll let you know 😃

JIC - trains still works after that, it's just that the new user is not added and hence is not able to login

works like a charm! you guys are the best, as always =)

original task name contains double space -> saved checkpoint also contains double space -> MODEL URL field in model description of this checkpoint in ClearML converts double space into single space. so when you copy & paste it somewhere, it'll be incorrect

yeah, I was thinking mainly about AWS. we use force to make sure we are using the correct latest checkpoint, but this increases costs when we are running a lot of experiments

yes

I'm not sure since names of these parameters do not match with boto3 names, and num_download_attempt is passed https://github.com/allegroai/clearml/blob/3d3a835435cc2f01ff19fe0a58a8d7db10fd2de2/clearml/storage/helper.py#L1439 as container.config.retries

maybe db somehow got corrupted ot smth like this? I'm clueless

1.3.0

{
username: "username"
password: "password"
name: "John Doe"
},

I guess I could edit docker-compose.yaml

python3 slack_alerts.py --channel trains-alerts --slack_api "OUR_KEY" --include_completed_experiments --include_manual_experiments

nope, same problem even after creating a new experiment from scratch

no, I even added the argument to specify tensorboard log_dir to make sure this is not happening

after the very first click, there is a popup with credentials request. nothing happens after that