How To Tell Clearml Server To Use Cloud Storage (Azure)? I Have A Clearml Server Deployed With Docker-Compose. As Per Instruction

Answered

How to tell clearML server to use cloud storage (Azure)?
I have a clearML server deployed with docker-compose.
As per instruction here , I added the file /opt/clearml/config/clearml.conf :

azure.storage {
    containers: [
        {
            account_name: "XXXXX"
            account_key: "XXXXXXX"
            container_name: "clearml"
        }
    ]
}

Restart my container and ... nothing change. I train a new model and nothing change in my Azure blob storage ...
In the docker stdout I can only see this for clearml-fileserver:

clearml-fileserver      | Loading config from /opt/clearml/fileserver/config/default                                                                                                                               
clearml-fileserver      | Loading config from file /opt/clearml/fileserver/config/default/fileserver.conf                                                                                                          
clearml-fileserver      | Loading config from file /opt/clearml/fileserver/config/default/logging.conf                                                                                                             
clearml-fileserver      | Loading config from /opt/clearml/config                                                                                                                                                  
clearml-fileserver      | Loading config from file /opt/clearml/config/clearml.conf                                                                                                                                
clearml-fileserver      |  * Serving Flask app "fileserver" (lazy loading)                                                                                                                                         
clearml-fileserver      |  * Environment: production                                                                                                                                                               
clearml-fileserver      |    WARNING: This is a development server. Do not use it in a production deployment.                                                                                                      
clearml-fileserver      |    Use a production WSGI server instead.                                                                                                                                                 
clearml-fileserver      |  * Debug mode: off

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Votes Newest

Answers 10

I didn;t know that from the client side, you can specify the storage elsewhere than the clearML server. Good to know !
But I still want to know, if possible, to use a blob storage by default, configured on the ClearML server, and each client don't need to do that ...

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Hi.
How do you tell the server to use my azure storage instead of local drive, on the host machine ? Isn't it by setting azure.storage in /opt/clearml/config/clearml.conf ?

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

@<1523701087100473344:profile|SuccessfulKoala55> Is it even possible to have the server storing file to a given blob storage ?

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

If so, that's not possible. You can however point the fileserver data folder to some mounted folder - if you have something that can create a mount from a filesystem folder to azure blob, it will work (the file server will always treat it as a local file system)

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

@<1576381444509405184:profile|ManiacalLizard2> do you mean having the ClearML FileServer store on azure blob instead of on the local drive?

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

When i set output uri in the client, artefact are sent to blob storage
When file_server is set to azure:// then model/checkpoint are sent to blob storage
But the are still plot and metrics folder that are stored in the server local disk. Is it correct?

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Just adding, that instead of using the fileserver and having it store data on Azure, you can simple configure your ClearML clients to store that data directly on Azure and not use the fileserver at all

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Hi @<1576381444509405184:profile|ManiacalLizard2> ,

In clearml, storage is handled on the client side. The fileserver on the server side is just another option, but does not determine what the clients will do.
The link you attached explains how to configure the clearml.conf file on the machine running the SDK, and you need to setup the output_uri option of Task.init() and/or to set the fielserver host url in your clearml.conf file to point to the storage you'd like to use (see None )

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

If you set the files_server in the config file to the blob storage, the rest will be stored there as well

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

do you mean having the ClearML FileServer store on azure blob instead of on the local drive?

Yes, that is what I wanted.

If so, that's not possible. You can however point the fileserver data folder to some mounted folder - if you have something that can create a mount from a filesystem folder to azure blob, it will work (the file server will always treat it as a local file system)

Thanks for confirming that it's the only solution. 👍

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Write your answer

782 Views

10 Answers

11 months ago