Hi @<1558986867771183104:profile|ShakyKangaroo32> , can you please share the logs from the async_delete docker container?
Is there a way to view the version in the web GUI?
WebApp: 1.9.1-312 • Server: 1.9.1-312 • API: 2.23
@<1558986867771183104:profile|ShakyKangaroo32> - if you check the api section in your client-side clearml.conf, the value for files_server there should be the same one that you set in the .env file on the server. Can you check that they are indeed the same?
If they are the same - can you please send me the output of the following command in the server:sudo docker logs -n30 async_delete
There is no chance of corrupting other experiments or databases?
I would backup the dbs prior to the upgrade so that you can rollback in case any issue arise in the upgrade process
As long as you delete only from the deleted tasks folders it should be OK
The 1.10 version handles files deletion differently so there is chance that it fixes the issue. If you use the default apiserver port then I would try upgrading. If you override the apiserver port then please wait for the hotfix version 1.10.1 that should be released soon
Hi @<1558986867771183104:profile|ShakyKangaroo32> , can you please open a GitHub issue to follow up on this? I think a fix should be issued shortly afterwards
Hi @<1558986867771183104:profile|ShakyKangaroo32> , what version of ClearML server are you using?
If you go to settings, the versions should appear at the bottom right
As a temporary solution, shutting down the entire docker-compose, deleting the left over files using administrator permissions and then bringing it back up again, does this sound reasonable?
@<1558986867771183104:profile|ShakyKangaroo32> - are you running the server using docker-compose?
If so - please add the following to the .env file in the same directory as the compose:CLEARML_FILES_HOST=http:// <YOUR IP or HOSTNAME> :8081
Then restart the service by running sudo docker-compose up -d
Please update if this worked, or if you have any questions/issues
This container may have not been introduced in this version yet, I don't see one in docker ps
@<1523703097560403968:profile|CumbersomeCormorant74> Hi, thanks for the suggestion, but unfortunately, it did not work.
I meant, as a temporary solution instead of upgrading
Do you think that updating to a new version will probably fix this?
The value is the same.
As I mentioned before, the server version I'm working with does not have the async_delete container. Unfortunately, due to internal considerations, the version update will not take place in the near future, so I am having the system admin delete them for me manually every once in a while.
Keeping the current version and deleting manually