Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi. I Somehow Managed To Exceed The Metrics Quota By ~35Gb. I Logged Some Histograms, But Still That Seems Excessive. Now I Am Trying To Delete Archived Experiments With The Cleanup Service, But Some Tasks Cannot Be Deleted:

Hi. I somehow managed to exceed the metrics quota by ~35GB. I logged some histograms, but still that seems excessive. Now I am trying to delete archived experiments with the cleanup service, but some tasks cannot be deleted:

Deleting 1ea58930d39441cf93ae63f1a1da92cd 2022-10-27 07:45:39,069 - clearml.Task - ERROR - Task deletion failed: unhashable type: 'dict' Deleting a256524c770043f38de66e14d820a6a1 2022-10-27 07:46:14,344 - clearml.Task - ERROR - Task deletion failed: unhashable type: 'dict' Deleting 80345bd6e27845deb13cc3f0fd4712be 2022-10-27 07:46:59,239 - clearml.Task - ERROR - Task deletion failed: unhashable type: 'dict' Deleting 2f6585ae2336467dac77b28a426686bc 2022-10-27 07:47:36,037 - clearml.Task - ERROR - Task deletion failed: unhashable type: 'dict' Deleting 3d9c9076f36941e2a8bf67ec6d3d2a92 2022-10-27 07:47:43,613 - clearml.Task - ERROR - Task deletion failed: unhashable type: 'dict' Deleting 66a7255a7bed4f149cbf919ec7e4ebf2 2022-10-27 07:48:46,018 - clearml.Task - ERROR - Task deletion failed: unhashable type: 'dict' Deleting 224b935360df497584401375e98ff33bThose are all tasks that include the histograms, so maybe that's related. Seems to be a server issue, at least I don't know what I can do on the client side. I'm using the app.clear.ml server.

  
  
Posted 2 years ago
Votes Newest

Answers 24


AgitatedDove14 the cleanup_service.py script in the repository, which https://github.com/allegroai/clearml/blob/ff7b174bf162347b82226f413040ff6473401e92/examples/services/cleanup/cleanup_service.py#L82 the snippet I posted.

  
  
Posted 2 years ago

Friendly ping CostlyOstrich36

  
  
Posted 2 years ago

AgitatedDove14 those are all tasks for which I have accidentally logged a large amount of histograms, in the order of gigabytes. It consistently fails when I try to delete the same task

  
  
Posted 2 years ago

Regarding the slowness, it does not download the files, but it will send a delete request for each one (which might be a considerable number of delete requests)

  
  
Posted 2 years ago

AgitatedDove14 CostlyOstrich36 Sorry for pinging again, but is there anything I can do to delete those tasks?

  
  
Posted 2 years ago

AgitatedDove14 could you maybe have a look? For some reason I am not able to delete some (particularly large) tasks using the cleanup service, i.e. API calls in the form

deleted_task = Task.get_task(task_id=task.id) deleted_task.delete( delete_artifacts_and_models=True, skip_models_used_by_other_tasks=True, raise_on_error=False )

  
  
Posted 2 years ago

SuccessfulKoala55 yes that gives some more information:

` Deleting 11 tasks
Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/reward-learner-ASq25l3C-py3.10/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 711, in _delete
[x for x in filter(None, self._get_image_plot_uris()) if not callback or callback("image_plot", x)]
File "/root/.cache/pypoetry/virtualenvs/reward-learner-ASq25l3C-py3.10/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 810, in _get_image_plot_uris
return self._get_all_events(event_type="plot", unique_selector=image_source_selector, batch_size=10000)
File "/root/.cache/pypoetry/virtualenvs/reward-learner-ASq25l3C-py3.10/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 2508, in _get_all_events
events_list.update(log_events.response.events)
TypeError: unhashable type: 'dict'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/container/cleanup.py", line 65, in <module>
deleted_task.delete(
File "/root/.cache/pypoetry/virtualenvs/reward-learner-ASq25l3C-py3.10/lib/python3.10/site-packages/clearml/task.py", line 1809, in delete
return super(Task, self)._delete(
File "/root/.cache/pypoetry/virtualenvs/reward-learner-ASq25l3C-py3.10/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 733, in _delete
raise self.DeleteError("Task deletion failed: {}".format(ex))
clearml.backend_interface.task.task.Task.DeleteError: Task deletion failed: unhashable type: 'dict' `

  
  
Posted 2 years ago

FlutteringWorm14 Can you change the delete() call to use the raise_on_error=True argument? This might print out a better stack trace...

  
  
Posted 2 years ago

Task deletion failed: unhashable type: 'dict'Hi FlutteringWorm14 trying to figure where this is coming from, give me a sec

  
  
Posted 2 years ago

using the cleanup service

Wait FlutteringWorm14 , the cleanup service , or task.delete call ? (these are not the same)

  
  
Posted 2 years ago

Let me take a look, what's the clearml-server version and clearml python version?

  
  
Posted 2 years ago

Hi FlutteringWorm14 , what happens if you try to delete those tasks manually?

  
  
Posted 2 years ago

Thanks :hugging_face:

  
  
Posted 2 years ago

Some run hashes are in the logs I posted, if you have the permissions to access these feel free

  
  
Posted 2 years ago

FlutteringWorm14 any insight on the Task the it fails to delete ? or to reproduce ?

  
  
Posted 2 years ago

Hi FlutteringWorm14 , yes, this explains a lot

  
  
Posted 2 years ago

The server is the public one hosted at http://app.clear.ml . The client is at version 1.7.2

  
  
Posted 2 years ago

Wait for it... 🙂

  
  
Posted 2 years ago

How can I delete them manually? Is that possible in the UI?

  
  
Posted 2 years ago

The update is done each morning

  
  
Posted 2 years ago

This works SuccessfulKoala55 ! It's very slow though, it's probably downloading the data before deleting it. But that's okay, at least it works. Thanks a lot 🙂

  
  
Posted 2 years ago

Do you know when the next update of the usage metrics is scheduled? Do I have to wait until tomorrow before I can use clearml again?

  
  
Posted 2 years ago

Try 1.7.3rc0 🙂

  
  
Posted 2 years ago

Great SuccessfulKoala55 🙂 Do you have any ideas on things I could try to work around the issue / further clarify it?

  
  
Posted 2 years ago