Hi FlutteringWorm14 , yes, this explains a lot
Hi FlutteringWorm14 , what happens if you try to delete those tasks manually?
AgitatedDove14 CostlyOstrich36 Sorry for pinging again, but is there anything I can do to delete those tasks?
The server is the public one hosted at http://app.clear.ml . The client is at version 1.7.2
SuccessfulKoala55 yes that gives some more information:
` Deleting 11 tasks
Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/reward-learner-ASq25l3C-py3.10/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 711, in _delete
[x for x in filter(None, self._get_image_plot_uris()) if not callback or callback("image_plot", x)]
File "/root/.cache/pypoetry/virtualenvs/reward-learner-ASq25l3C-py3.10/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 810, in _get_image_plot_uris
return self._get_all_events(event_type="plot", unique_selector=image_source_selector, batch_size=10000)
File "/root/.cache/pypoetry/virtualenvs/reward-learner-ASq25l3C-py3.10/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 2508, in _get_all_events
events_list.update(log_events.response.events)
TypeError: unhashable type: 'dict'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/container/cleanup.py", line 65, in <module>
deleted_task.delete(
File "/root/.cache/pypoetry/virtualenvs/reward-learner-ASq25l3C-py3.10/lib/python3.10/site-packages/clearml/task.py", line 1809, in delete
return super(Task, self)._delete(
File "/root/.cache/pypoetry/virtualenvs/reward-learner-ASq25l3C-py3.10/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 733, in _delete
raise self.DeleteError("Task deletion failed: {}".format(ex))
clearml.backend_interface.task.task.Task.DeleteError: Task deletion failed: unhashable type: 'dict' `
FlutteringWorm14 Can you change the delete() call to use the raise_on_error=True argument? This might print out a better stack trace...
How can I delete them manually? Is that possible in the UI?
Regarding the slowness, it does not download the files, but it will send a delete request for each one (which might be a considerable number of delete requests)
Task deletion failed: unhashable type: 'dict'Hi FlutteringWorm14 trying to figure where this is coming from, give me a sec
Great SuccessfulKoala55 🙂 Do you have any ideas on things I could try to work around the issue / further clarify it?
Do you know when the next update of the usage metrics is scheduled? Do I have to wait until tomorrow before I can use clearml again?
AgitatedDove14 those are all tasks for which I have accidentally logged a large amount of histograms, in the order of gigabytes. It consistently fails when I try to delete the same task
Some run hashes are in the logs I posted, if you have the permissions to access these feel free
Let me take a look, what's the clearml-server version and clearml python version?
using the cleanup service
Wait FlutteringWorm14 , the cleanup service , or task.delete call ? (these are not the same)
AgitatedDove14 the cleanup_service.py script in the repository, which https://github.com/allegroai/clearml/blob/ff7b174bf162347b82226f413040ff6473401e92/examples/services/cleanup/cleanup_service.py#L82 the snippet I posted.
FlutteringWorm14 any insight on the Task the it fails to delete ? or to reproduce ?
This works SuccessfulKoala55 ! It's very slow though, it's probably downloading the data before deleting it. But that's okay, at least it works. Thanks a lot 🙂
AgitatedDove14 could you maybe have a look? For some reason I am not able to delete some (particularly large) tasks using the cleanup service, i.e. API calls in the form
deleted_task = Task.get_task(task_id=task.id) deleted_task.delete( delete_artifacts_and_models=True, skip_models_used_by_other_tasks=True, raise_on_error=False )