I Have Setup A

Answered

I Have Setup A

I have setup a clearml-server running on a Azure VM instance and have used default parameters when it comes to specifying storage locations for data and artefacts. I have managed to create a versioned dataset of the images dataset I am using, by using clearml.Dataset to push these to the clearml-server .

I have now setup an Azure Blob Store container for use with the server, such that all artefacts (model weights etc) and datasets are stored on this blob store.

Q. Would someone mind outlining what the steps are to configuring the default storage locations, such that any artefacts or data which are pushed to the server are stored by default on the Azure Blob Store?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					VivaciousPenguin66
				
					0
					 × 1

Votes Newest

Answers 5

I suppose the same would need to be done for any

client

PC running

clearml

such that you are submitting dataset upload jobs?

Correct

That is, the dataset is perhaps local to my laptop, or on a development VM that is not in the

clearml

system, but I from there I want to submit a copy of a dataset, then I would need to configure the storage section in the same way as well?

Correct

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

I assume the account name and key refers to the storage account credentials that you can from Azure Storage Explorer?

correct

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

AgitatedDove14 Thanks for that.
I suppose the same would need to be done for any client PC running clearml such that you are submitting dataset upload jobs?

That is, the dataset is perhaps local to my laptop, or on a development VM that is not in the clearml system, but I from there I want to submit a copy of a dataset, then I would need to configure the storage section in the same way as well?

I assume the account name and key refers to the storage account credentials that you can from Azure Storage Explorer?
It gives you two access keys, primary and secondary from the properties tab of the specified storage account.

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					VivaciousPenguin66
				
					0
					 × 1

I am bit confused because I can see configuration sections Azure storage in the clearml.conf files, but these are on the client pc and the clearml-agent compute nodes.

So do these parameters have to be set on the clients and compute nodes individually, or is something that can be set on the server?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					VivaciousPenguin66
				
					0
					 × 1

Q. Would someone mind outlining what the steps are to configuring the default storage locations, such that any artefacts or data which are pushed to the server are stored by default on the Azure Blob Store?

Hi VivaciousPenguin66
See my reply here on configuring the default output uri on the agent: https://clearml.slack.com/archives/CTK20V944/p1621603564139700?thread_ts=1621600028.135500&cid=CTK20V944
Regrading permission setup:
You need to make sure you have the Azure blob credentials on the agent's machine, so it can actually access the blob storage. Make sense ?
https://github.com/allegroai/clearml-agent/blob/e93384b99bdfd72a54cf2b68b3991b145b504b79/docs/clearml.conf#L276

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Write your answer

961 Views

5 Answers

3 years ago

one year ago