I Buried This Issue In Another Thread To Do With Deployment, But I Was Wondering If Anyone Else Has Had Problems Using

Unanswered

Crawls out from under the table and takes a deep breath

AgitatedDove14 you remember we talked about it being a bug or a stupid.....

Well, it's a stupid by me.... somehow I managed to propagate irregularities in the clearml.conf file such that it successfully loaded, but the expected nested structure was not there.

When the get_local_copy() method requested the model, it correctly got the azure credentials, however when the StorageHelper class tries to get the azure credentials from the configuration file, it returns an empty dictionary structure, because of the mishaped conf file. The line question in the StorageHelper class is [ https://github.com/allegroai/clearml/blob/332ceab3eadef4997e897d171957975a247a6dc1/clearml/storage/helper.py#L195 ]:

_azure_configurations = AzureContainerConfigurations.from_config(config.get('azure.storage', {}))

Thus, when the get_config_by_uri method returns this empty structure in the _resolve_base_url method, it is then reported as Azure Credentials are not found. This is because the conf file had been corrupted in shape and thus when it tried to find the azure.storage fields, it doesn't find them.

Perhaps what is needed is some checking of the clearml.conf file to make sure that at least the basic fields are present and complain if not?

So, I have rerun clearml-agent init and transposed over my custom configurations and then run the StorageManager.get_local_copy(uri) method using the original URI, and bingo!:

Crawls out from under the table and takes a deep breath

AgitatedDove14 you remember we talked about it being a bug or a stupid.....

Well, it's a stupid by me.... somehow I managed to propagate irregularities in the clearml.conf file such that it successfully loaded, but the expected nested structure was not there.

_azure_configurations = AzureContainerConfigurations.from_config(config.get('azure.storage', {}))

Perhaps what is needed is some checking of the clearml.conf file to make sure that at least the basic fields are present and complain if not?

So, I have rerun clearml-agent init and transposed over my custom configurations and then run the StorageManager.get_local_copy(uri) method using the original URI, and bingo!:

2021-06-10 13:23:42,110 - clearml.storage - INFO - Downloading: 5.00MB / 81.72MB @ 20.34MBs from Birds%2FTraining/TRAIN [Network%3A resnet34, Library%3A torchvision] Ignite Train PyTorch CNN on CUB200.8611ada5be6f4bb6ba09cf730ecd2253/models/cub200_resnet34_ignite_best_model_0.pt 2021-06-10 13:23:42,346 - clearml.storage - INFO - Downloading: 13.00MB / 81.72MB @ 33.93MBs from Birds%2FTraining/TRAIN [Network%3A resnet34, Library%3A torchvision] Ignite Train PyTorch CNN on CUB200.8611ada5be6f4bb6ba09cf730ecd2253/models/cub200_resnet34_ignite_best_model_0.pt 2021-06-10 13:23:42,379 - clearml.storage - INFO - Downloading: 21.00MB / 81.72MB @ 243.96MBs from Birds%2FTraining/TRAIN [Network%3A resnet34, Library%3A torchvision] Ignite Train PyTorch CNN on CUB200.8611ada5be6f4bb6ba09cf730ecd2253/models/cub200_resnet34_ignite_best_model_0.pt 2021-06-10 13:23:42,537 - clearml.storage - INFO - Downloading: 29.00MB / 81.72MB @ 50.48MBs from Birds%2FTraining/TRAIN [Network%3A resnet34, Library%3A torchvision] Ignite Train PyTorch CNN on CUB200.8611ada5be6f4bb6ba09cf730ecd2253/models/cub200_resnet34_ignite_best_model_0.pt 2021-06-10 13:23:42,582 - clearml.storage - INFO - Downloading: 37.00MB / 81.72MB @ 176.97MBs from Birds%2FTraining/TRAIN [Network%3A resnet34, Library%3A torchvision] Ignite Train PyTorch CNN on CUB200.8611ada5be6f4bb6ba09cf730ecd2253/models/cub200_resnet34_ignite_best_model_0.pt 2021-06-10 13:23:42,801 - clearml.storage - INFO - Downloading: 45.00MB / 81.72MB @ 36.65MBs from Birds%2FTraining/TRAIN [Network%3A resnet34, Library%3A torchvision] Ignite Train PyTorch CNN on CUB200.8611ada5be6f4bb6ba09cf730ecd2253/models/cub200_resnet34_ignite_best_model_0.pt 2021-06-10 13:23:42,848 - clearml.storage - INFO - Downloading: 53.00MB / 81.72MB @ 168.57MBs from Birds%2FTraining/TRAIN [Network%3A resnet34, Library%3A torchvision] Ignite Train PyTorch CNN on CUB200.8611ada5be6f4bb6ba09cf730ecd2253/models/cub200_resnet34_ignite_best_model_0.pt 2021-06-10 13:23:42,893 - clearml.storage - INFO - Downloading: 61.00MB / 81.72MB @ 179.29MBs from Birds%2FTraining/TRAIN [Network%3A resnet34, Library%3A torchvision] Ignite Train PyTorch CNN on CUB200.8611ada5be6f4bb6ba09cf730ecd2253/models/cub200_resnet34_ignite_best_model_0.pt 2021-06-10 13:23:43,082 - clearml.storage - INFO - Downloading: 69.72MB / 81.72MB @ 46.13MBs from Birds%2FTraining/TRAIN [Network%3A resnet34, Library%3A torchvision] Ignite Train PyTorch CNN on CUB200.8611ada5be6f4bb6ba09cf730ecd2253/models/cub200_resnet34_ignite_best_model_0.pt 2021-06-10 13:23:43,412 - clearml.storage - INFO - Downloading: 77.72MB / 81.72MB @ 24.21MBs from Birds%2FTraining/TRAIN [Network%3A resnet34, Library%3A torchvision] Ignite Train PyTorch CNN on CUB200.8611ada5be6f4bb6ba09cf730ecd2253/models/cub200_resnet34_ignite_best_model_0.pt 2021-06-10 13:23:43,450 - clearml.storage - INFO - Downloaded 81.72 MB successfully from Birds%2FTraining/TRAIN [Network%3A resnet34, Library%3A torchvision] Ignite Train PyTorch CNN on CUB200.8611ada5be6f4bb6ba09cf730ecd2253/models/cub200_resnet34_ignite_best_model_0.pt , saved to /home/edmorris/.clearml/cache/storage_manager/global/e38f6052e6b887337635fc2821a6b5d4.cub200_resnet34_ignite_best_model_0.pt '/home/edmorris/.clearml/cache/storage_manager/global/e38f6052e6b887337635fc2821a6b5d4.cub200_resnet34_ignite_best_model_0.pt'

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					VivaciousPenguin66
				
					0
					 × 1

151 Views

0 Answers

3 years ago

one year ago