FiercePenguin76 does this happen in a self-hosted ClearML Server? If so, what is the server version?
this does not prevent from enqueuing and running new tasks, rather an eyesore
for the tasks that are not deleted, log is different:[2021-09-09 12:19:07,718] [8] [WARNING] [clearml.service_repo] Returned 400 for tasks.dequeue in 4ms, msg=Invalid task id: status=stopped, expected=queued
but the old ones are there, and I can’t do anything about them
self-hosted. Just upgraded to latest version today (1.1.1). The problem appeared when we were still using 1.0.2
log:[2021-09-09 11:22:09,339] [8] [WARNING] [clearml.service_repo] Returned 400 for tasks.dequeue in 2ms, msg=Invalid task id: id=28d2cf5233fe41399c255950aa8b 8c9d,company=d1bd92a3b039400cbafc60a7a5b1e52b
FiercePenguin76 , we'll try to reproduce. Thanks for the input!
no new unremovable entries have appeared (although I haven’t tried)
I think they appeared when I had a lot of HPO tasks enqueued and not started yet, and then I decided to either Abort or Archive them - I don’t remember already