Hi, I’Ve Set Up A Clearml Server With The Default_Output_Uri Pointed To S3. We’Re Planning To Migrate From S3 To Azure Blob Storage. Is There A Direct Way To Migrate The Data, Or Should We Simply Transfer The Data From S3 To Azure And Update The Default_

Answered

Hi,
I’ve set up a ClearML server with the default_output_uri pointed to S3. We’re planning to migrate from S3 to Azure Blob Storage.

Is there a direct way to migrate the data, or should we simply transfer the data from S3 to Azure and update the default_output_uri to the new Azure Blob Storage path?

Thanks in advance

  				
Posted 
	9 months ago

					More  		
  Report
		
					QuaintJellyfish58
				
					0
					 × 1

Votes Newest

Answers 4

Hi SuccessfulKoala55 ,
We have successfully created a sample for the migration. Here are the changes:

URL for MongoDB from s3:// to azure://
Elasticsearch as you suggested
However, our main focus is that most of our production fetches models from ClearML, which are configured with s3:// URLs.

There is an issue/bug in the UI when downloading via Azure. Here are the details: None .

  				
Posted 
	8 months ago

					More  		
  Report
		
					QuaintJellyfish58
				
					0
					 × 1

Thanks guys, i will try to learn that first. i will updates when executing these ideas. SuccessfulKoala55 AmiableSeaturtle81 🙌

  				
Posted 
	9 months ago

					More  		
  Report
		
					QuaintJellyfish58
				
					0
					 × 1

We had a similar problem. Clearml doesnt support data migration (not that I know of)
So you have two ways to fix this:

Recreate the dataset when its already in Azure
Edit each elasticsearch database file entry to point to new destination (we did this)

  				
Posted 
	9 months ago

					More  		
  Report
		
					AmiableSeaturtle81
				
					0
					 × 1

Hi QuaintJellyfish58 , see here for an example on how to accomplish that: None

  				
Posted 
	9 months ago

					More  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Write your answer

949 Views

4 Answers

9 months ago

8 months ago