Hi @<1523701868901961728:profile|ReassuredTiger98> , deleting debug images is supported when using the SDK to delete experiments, and can also be supported on the server side (e.g. when delete requests are sent from the UI) - for the latter you'll need to make sure the server has the correct credentials configured under the storage_credentials section in the services.conf server configuration file which you can add un the clearml server configuration folder. See here for more details.
Hi @<1523701868901961728:profile|ReassuredTiger98> , how exactly do you override the values in storage_credentials file? Do you prepare a new docker image with the changed file or map this file from outside with the volume mapping in the docker compose or through the env variables? What is also important is that you do this override for the async_delete service. It is the service that actually uses the storage credentials. Not the apiserver itself
@<1523701994743664640:profile|AppetizingMouse58> Thank you very much. I forgot the volume mapping.
So can I just add the config to the async_delete container and mirror the directory structure from github?
volumes:
- /opt/clearml/config:/opt/clearml/config
- /opt/clearml/logs:/var/log/clearml
And the files that I see on github are the default configuration of the server, even if I do not have these files in my installation, right?
Correct
Hi @<1523701087100473344:profile|SuccessfulKoala55> Thank you very much.
Is there some way to verify the server uses the correct configuration files? (E.g. see it in the logs/web ui). I Just tried it does not work.
At least I can see the async_delete service complains about a missing secret, so I can start debugging there. I am using the same config as for my agents, but somehow for async_delete it does not work...
And the files that I see on github are the default configuration of the server, even if I do not have these files in my installation, right?
I just wanna avoid that ClearML leaves files lingering around. Btw: a better default behavior in my opinion would be to delete tasks only after files have been deleted. And only with the force option to delete the task anyways!