Hi All, I’M Trying To Work Out How To Reduce The Amount Of Space Taken Up By Old Tasks On Our Self Hosted Server. I’Ve Deleted About 400 Tasks (200 Or So Were Already Archived) Out Of About 2000 And Was Expecting To See A Reduction In Folder Size In

Answered

Hi all, I’m trying to work out how to reduce the amount of space taken up by old tasks on our self hosted server. I’ve deleted about 400 tasks (200 or so were already archived) out of about 2000 and was expecting to see a reduction in folder size in /opt/clearml/data , but it’s exactly the same! Am I right to expect the folder size to reduce? Is there a way of working out which tasks are taking up the most storage?

For context, all artifacts are stored separately and the majority of the disk use is in /opt/clearml/data/elastic_7

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuperiorPanda77
				
					0
					 × 1

Votes Newest

Answers 4

Shards that I can see are using a lot of disk space are

events-training_stats_scalar
events-log
And then various worker_stats_*

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuperiorPanda77
				
					0
					 × 1

Hi @<1523701092473376768:profile|SuperiorPanda77> , while data in ES is deleted, compacting the indices is not something ES does immediately - you can look in the EE documentation in how to trigger that yourself (can't look for it now)

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Thanks @<1523701087100473344:profile|SuccessfulKoala55> , I’ve taken a look and is this force merging you’re referring to? Do you know how often ES is configured to merge in clearml server?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuperiorPanda77
				
					0
					 × 1

I don't think we configure it in any special way, so the default is probably used

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Write your answer

998 Views

4 Answers

one year ago