Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, We Have A Self Hosted Clearml Server Which I Mainly Use For Experiment Tracking. There Is One Issue I Have Noticed Recently: Whenever I Archive And Delete An Experiment (With The Box " Remove All Related Artifacts And Debug Samples From Clearml File

Hi, we have a self hosted ClearML server which I mainly use for experiment tracking.

There is one issue I have noticed recently: whenever I archive and delete an experiment (with the box " Remove all related artifacts and debug samples from ClearML file server" checked), and then examine the actual directories in the file server, the directory corresponding to that experiment still exists, along with the previously saved model weights and debug samples.

Is this the expected behavior? should I manually clear those files with root permissions?

  
  
Posted 12 months ago
Votes Newest

Answers 20


@<1558986867771183104:profile|ShakyKangaroo32> - if you check the api section in your client-side clearml.conf, the value for files_server there should be the same one that you set in the .env file on the server. Can you check that they are indeed the same?
If they are the same - can you please send me the output of the following command in the server:
sudo docker logs -n30 async_delete

  
  
Posted 12 months ago

Hi @<1558986867771183104:profile|ShakyKangaroo32> , can you please share the logs from the async_delete docker container?

  
  
Posted 12 months ago

This container may have not been introduced in this version yet, I don't see one in docker ps

  
  
Posted 12 months ago

The 1.10 version handles files deletion differently so there is chance that it fixes the issue. If you use the default apiserver port then I would try upgrading. If you override the apiserver port then please wait for the hotfix version 1.10.1 that should be released soon

  
  
Posted 12 months ago

As a temporary solution, shutting down the entire docker-compose, deleting the left over files using administrator permissions and then bringing it back up again, does this sound reasonable?

  
  
Posted 12 months ago

Do you think that updating to a new version will probably fix this?

  
  
Posted 12 months ago

I would backup the dbs prior to the upgrade so that you can rollback in case any issue arise in the upgrade process

  
  
Posted 12 months ago

There is no chance of corrupting other experiments or databases?

  
  
Posted 12 months ago

I meant, as a temporary solution instead of upgrading

  
  
Posted 12 months ago

Hi @<1558986867771183104:profile|ShakyKangaroo32> , can you please open a GitHub issue to follow up on this? I think a fix should be issued shortly afterwards

  
  
Posted 12 months ago

As long as you delete only from the deleted tasks folders it should be OK

  
  
Posted 12 months ago

Keeping the current version and deleting manually

  
  
Posted 12 months ago

Great. Thanks!

  
  
Posted 12 months ago

If you go to settings, the versions should appear at the bottom right

  
  
Posted 12 months ago

Is there a way to view the version in the web GUI?

  
  
Posted 12 months ago

WebApp: 1.9.1-312 • Server: 1.9.1-312 • API: 2.23

  
  
Posted 12 months ago

@<1558986867771183104:profile|ShakyKangaroo32> - are you running the server using docker-compose?
If so - please add the following to the .env file in the same directory as the compose:
CLEARML_FILES_HOST=http:// <YOUR IP or HOSTNAME> :8081
Then restart the service by running sudo docker-compose up -d
Please update if this worked, or if you have any questions/issues

  
  
Posted 12 months ago

The value is the same.
As I mentioned before, the server version I'm working with does not have the async_delete container. Unfortunately, due to internal considerations, the version update will not take place in the near future, so I am having the system admin delete them for me manually every once in a while.

  
  
Posted 12 months ago

@<1523703097560403968:profile|CumbersomeCormorant74> Hi, thanks for the suggestion, but unfortunately, it did not work.

  
  
Posted 12 months ago

Hi @<1558986867771183104:profile|ShakyKangaroo32> , what version of ClearML server are you using?

  
  
Posted 12 months ago
605 Views
20 Answers
12 months ago
12 months ago
Tags