Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Am A Newbie Here And I Am Just Struggling With Some Chanllenges When I Self-Host The Clearml Server ( In Linux - Ubuntu 20.04 )

hi, i am a newbie here and i am just struggling with some chanllenges when i self-host the ClearML Server ( in Linux - Ubuntu 20.04 )

Background :
I successfully downloaded and ran docker-compose.yml, and ClearML Server has been established. Then, I successfully created, added, and uploaded datasets to ClearML's fileserver via cli and i wanted to delete the datasets via WebUI.

Problem :
I first archived the dataset and selected the Delete Action in the WebUI. Then the datasets on the WebUI disappeared, but I checked the local directory and found that the files were not deleted from the clearml-fileserver( built by clearml-fileserver image ). It seems that only some information in the database was deleted but not files.

For example, I can still see the dataset in the local directory:

/opt/clearml/data/fileserver/<project_name>/.datasets/<dataset_name>/<files>

After encountering this problem, i try to trace back what was happening :

  • I tried to check the Network Status in the Browser (F12), and found a triggered api called tasks.delete_many, it return 200 status code, and data ( incuding some uri like : None :port/<project>/.datasets/<dataset_name>/<files> )
  • Then i tried to use curl -X DELETE <uri above> , it successfully deleted the files in the local directory (/opt/clearml/data/fileserver/........) , also i checked the logs in clearml-fileserver container , it show that delete event was triggered when i sent the delete restful api.
  • I tried to create, add, upload datasets to fileserver and tried to delete it via WebUI again. After clicking the Delete Button, the Newtwork Status show that tasks.delete_many was triggered, but i can't see any event recorded in the clearml-server's logs.
    I suspect there may be some issues with the tasks.delete_many API, as the datasets was not deleted correctly but returned a 200 status code. Moreover, this API seems to be expected to do the following: curl -X DELETE . Based on the API result, it did return the correct URI, but it seems that curl -X DELETE was not triggered. Should I set something like async_delete to synchronize the deletion task on the file server? (But async_delete should only be set for external storage, right? I am using clearml-fileserver and running it on None locally.)
  
  
Posted 2 months ago
Votes Newest

Answers 6


result in 【 tasks.delete_many API 】

"meta": {
"id": "c836e23d3308421fa8eb84e67217b0dc",
"trx": "c836e23d3308421fa8eb84e67217b0dc",
"endpoint": {
"name": "tasks.delete_many",
"requested_version": "2.28",
"actual_version": "1.0"
},
"result_code": 200,
"result_subcode": 0,
"result_msg": "OK",
"error_stack": "",
"error_data": {}
},
"data": {
"succeeded": [
{
"id": "127388d22df945d2a0ab59b47a7ebca9",
"deleted": true,
"updated_children": 0,
"updated_models": 0,
"deleted_models": 0,
"urls": {
"model_urls": [],"event_urls": [],
"artifact_urls": [
" http://myip:8081/1234/upload.127388d22df945d2a0ab59b47a7ebca9/artifacts/conf/data.zip "
]
}
}
],
"failed": []
}

  
  
Posted 2 months ago

Cany anyone help me 🥲

It seems like WebUI didn't trigger the delete event ( fileserver ) but only delete the information recorded in mongo ( like datasets/artifacts name , version...... etc )

  
  
Posted 2 months ago

I can do anything ( create,add,upload,delete,publish ) project/datasets successfully using CLI ( including information in DB and files in the local directory ) , However when I delete them via WebUI, all files can't not be removed from the local directory.

  
  
Posted 2 months ago

How did you verify that the files are still there? Not just you can see the folder name itself but are there contents? If you use the same links through the web UI, can you still download files even when task/dataset is deleted?

  
  
Posted 2 months ago

Hi @<1702130048917573632:profile|BlushingHedgehong95> , I would suggest the following few tests:

  • Run some mock task that uploads an artifact to the files server. Once done, verify you can download the artifact via the web UI - there should be a link to it. Save that link. Then delete the task and mark to delete all artifacts. Test the link again to see that it fails to delete
  • Please repeat the same with a dataset
  
  
Posted 2 months ago

@<1523701070390366208:profile|CostlyOstrich36> I just use the command to check the folder in the terminal ( both ls
And du -h <folder> ) , it shows that the datasets are still here! Also, I can still download file even when it is deleted via WebUI.

I guess there is some method to delete the files in the self-hosted clearml-fileserver like async_delete? 🥲

  
  
Posted 2 months ago
221 Views
6 Answers
2 months ago
2 months ago
Tags
Similar posts