Hi Guys. I'M Struggling To Get The Cleanup Service Working On Our On-Prem Setup. We Are Using The Built In Service (

Answered

Hi guys.

I'm struggling to get the Cleanup Service working on our on-prem setup. We are using the built in service ( None ) but see loads of errors like:

Could not delete Task ID=bd0f6aaf0cfc41c9a1a5db99ea417be7, 'Task' object has no attribute 'delete'

When I run the script manually and try to delete just a singe Task I get the following:

2024-01-05 09:52:30,202 - clearml.Task - ERROR - Action failed <400/201: models.delete/v1.0 (Invalid model id: company=d1bd92a3b039400cbafc60a7a5b1e52b, id=25555cba30334f359a6b6ff29b1c23cb)> (model=25555cba30334f359a6b6ff29b1c23cb, force=False)
...

I've checked system permissions on the server and it looks fine. What am I missing? We're running out of server space fast 🙂

Here's a gist of a modified cleanup_service script that just tries to delete a single task:
None

The server has the following versions WebApp: 1.11.0-373 • Server: 1.11.0-373 • API: 2.25 and we use ClearML version 1.11.1

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					GiganticMole91
				
					0
					 × 1

Votes Newest

Answers 4

Hi @<1523701601770934272:profile|GiganticMole91> , I've just tried the same thing (calls identical to your script) and it works for me, I'm trying to think maybe it has something to do with your models? Can you locate a model with the ID shown in the error?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Hi @<1523701087100473344:profile|SuccessfulKoala55> , thanks for responding. I've found out that my first error came from cloning a super old version of the clean up task in the web UI 😄
I don't know about the other error, to me it looks like the task gets deleted before handling errors, but since an error occurred (some 404 stuff, maybe the files actually aren't there) when deleting some artifacts on the task, clearml tries to reload the task and fails, with the 400/201 or 400/101. Something like this:

Deleting

 with type artifact
2024-01-05 14:08:14,675 - clearml.storage - WARNING - Failed deleting object /myproject/testing/alternating-stick.5deba388a0a1416f8b24e760813f4080/artifacts/config-file/config.yaml (404): File /mnt/fileserver/myproject/testing/alternating-stick.5deba388a0a1416f8b24e760813f4080/artifacts/config-file/config.yaml not found
2024-01-05 14:08:14,679 - clearml.Task - ERROR - Failed deleting the following URIs:


2024-01-05 14:08:14,703 - clearml.Task - ERROR - Action failed <400/101: tasks.get_by_id/v1.0 (Invalid task id: id=589f573a250a4af4acf016762deb5141, company=d1bd92a3b039400cbafc60a7a5b1e52b)> (task=589f573a250a4af4acf016762deb5141)
2024-01-05 14:08:14,706 - clearml.Task - ERROR - Failed reloading task 589f573a250a4af4acf016762deb5141
2024-01-05 14:08:14,725 - clearml.Task - ERROR - Action failed <400/101: tasks.get_by_id/v1.0 (Invalid task id: id=589f573a250a4af4acf016762deb5141, company=d1bd92a3b039400cbafc60a7a5b1e52b)> (task=589f573a250a4af4acf016762deb5141)
2024-01-05 14:08:14,726 - clearml.Task - ERROR - Failed reloading task 589f573a250a4af4acf016762deb5141

Does that sound plausible to you?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					GiganticMole91
				
					0
					 × 1

It might, yes, the question is can you reproduce it?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Sorry, I got caught up by other tasks. I might investigate further later, but it's not top of mind right now. Our main issue is to get people to archive their old tasks and models so they can be cleaned up 😄

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					GiganticMole91
				
					0
					 × 1

Write your answer

720 Views

4 Answers

10 months ago