Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Unanswered
Dear Clearml Community, I Am Trying To Optimize Storage On My Clearml File Server When Doing A Lot Of Experiments. To Achieve This, I Already Upload Only The Newest And Best Checkpoints To Clearml File Server Instead Of All Checkpoints. Another Component


Hello @<1523701205467926528:profile|AgitatedDove14> ,

Good news! It seems that using the list of URLs retrieved via "POST /events.get_task_events" and then deleting the corresponding images using StorageHelper class effectively does the trick! 🏆 FYI, here is the little function I wrote for deleting the files:

    from clearml.storage.helper import StorageHelper
    
    def delete_image_from_clearml_server(image_url: str) -> None:
        storage_helper = StorageHelper.get(url=image_url)
        try:
            storage_helper.delete(path=image_url)
        except Exception as e:
            raise ValueError(f"Could not remove image with URL '{image_url}': {e}")

Now, even if this function with StorageHelper works fine, I observed that some images are not referenced in "/events.get_task_events" event list (even if I wait some time) 😥 . In fact, they physically exist on the server, but no URL points towards those. This implies that those few images are not detected and hence not deleted.

Are you aware of a limitation of "/events.get_task_events" preventing from fetching some of the images stored on the server? 🤔

Here is a picture and a video illustrating the fact that 18 images were effectively deleted, but 2 have not been listed in the events of "/events.get_task_events" and were hence not deleted...

Thank you very much for your insight!
Have a nice weekend 😉
image

  
  
Posted 9 months ago
102 Views
0 Answers
9 months ago
9 months ago