ClearML FAQ | Hi Everyone, I Have Been Using Clearml Self Hosted For Experiment Visualisation - Logging Loss And Uploading Debug Samples Etc. Our Clearml Server Is Running On A Aws Instance, With The Fileserver And Databases (Mongo, Redis Etc) On That Instance. We Run

Answered

Hi Everyone, I Have Been Using Clearml Self Hosted For Experiment Visualisation - Logging Loss And Uploading Debug Samples Etc. Our Clearml Server Is Running On A Aws Instance, With The Fileserver And Databases (Mongo, Redis Etc) On That Instance. We Run

Hi Everyone,

I have been using ClearML self hosted for experiment visualisation - logging loss and uploading debug samples etc.
Our ClearML server is running on a AWS instance, with the fileserver and databases (mongo, redis etc) on that instance.
We run a lot of experiments and log a lot of data, and I would like to move the fileserver to S3, and potentially move the databases off the instance onto something that scales with the size of the data so I don't need to monitor any disk usage.
Has anyone got any tips or guides on how to set this up?

Thanks in advance :)

  				
Posted 
	one year ago

					More
				  		
  Report
		
					TimelyCrab1
				
					0

Votes Newest

Answers 2

Hi TimelyCrab1 , directing all your outputs to s3 is actually pretty easy. You simply need to configure api.files_server: <S3_BUCKET/SOME_DIR> in clearml.conf of all machines working on it.

Migrating existing data is more difficult since everywhere in the system everything is saved as links. I guess you could change the links in mongodb but I would advise against it.

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

TimelyCrab1 there were several discussions here and in the GitHub issues on how to migrate fileserver data (databases should be just fine if you just move or backup/restore the data)

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Write your answer

600 Views

2 Answers

one year ago