Hi @<1558986867771183104:profile|ShakyKangaroo32> , what version of ClearML server are you using?
Is there a way to view the version in the web GUI?
If you go to settings, the versions should appear at the bottom right
WebApp: 1.9.1-312 • Server: 1.9.1-312 • API: 2.23
Hi @<1558986867771183104:profile|ShakyKangaroo32> , can you please share the logs from the async_delete docker container?
This container may have not been introduced in this version yet, I don't see one in docker ps
Do you think that updating to a new version will probably fix this?
The 1.10 version handles files deletion differently so there is chance that it fixes the issue. If you use the default apiserver port then I would try upgrading. If you override the apiserver port then please wait for the hotfix version 1.10.1 that should be released soon
As a temporary solution, shutting down the entire docker-compose, deleting the left over files using administrator permissions and then bringing it back up again, does this sound reasonable?
There is no chance of corrupting other experiments or databases?
I would backup the dbs prior to the upgrade so that you can rollback in case any issue arise in the upgrade process
I meant, as a temporary solution instead of upgrading
Keeping the current version and deleting manually
As long as you delete only from the deleted tasks folders it should be OK
Hi @<1558986867771183104:profile|ShakyKangaroo32> , can you please open a GitHub issue to follow up on this? I think a fix should be issued shortly afterwards
@<1558986867771183104:profile|ShakyKangaroo32> - are you running the server using docker-compose?
If so - please add the following to the .env file in the same directory as the compose:CLEARML_FILES_HOST=http:// <YOUR IP or HOSTNAME> :8081
Then restart the service by running sudo docker-compose up -d
Please update if this worked, or if you have any questions/issues
@<1523703097560403968:profile|CumbersomeCormorant74> Hi, thanks for the suggestion, but unfortunately, it did not work.
@<1558986867771183104:profile|ShakyKangaroo32> - if you check the api section in your client-side clearml.conf, the value for files_server there should be the same one that you set in the .env file on the server. Can you check that they are indeed the same?
If they are the same - can you please send me the output of the following command in the server:sudo docker logs -n30 async_delete
The value is the same.
As I mentioned before, the server version I'm working with does not have the async_delete container. Unfortunately, due to internal considerations, the version update will not take place in the near future, so I am having the system admin delete them for me manually every once in a while.