I Am Using Pytorch Lightning With Ddp Accelerator On 4 Gpus, And I Found Every Checkpoint Is Recorded 4 Times On Web Ui With Different Ids. One Is On

Answered

I am using pytorch lightning with ddp accelerator on 4 gpus, and I found every checkpoint is recorded 4 times on web UI with different ids. One is on default_output_uri ( s3://...) and three are on file///... . Is this a feature or bug?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					DefeatedOstrich93
				
					0
					 × 1

Votes Newest

Answers 7

this is pretty weird. PL should only save from rank==0 :
https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/connectors/checkpoint_connector.py#L394

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					ExcitedFish86
				
					0
					 × 1

DefeatedOstrich93 can you verify lightning actually only stored once ?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Thanks for the report!

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					GrumpyPenguin23
				
					0
					 × 1

see https://github.com/allegroai/clearml/issues/292

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					DefeatedOstrich93
				
					0
					 × 1

ClearML thanks

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					GrumpyPenguin23
				
					0
					 × 1

No running duplicate exps. Which repo to report? clearml-agent, clearml or clearml-server.

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					DefeatedOstrich93
				
					0
					 × 1

Hi! Looks like all the processes are calling torch.save so it's probably reflecting what Lightning did behind the curtain. Definitely not a feature though. Do you mind reporting this to our github repo? Also, are you also getting duplicate experiments?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					GrumpyPenguin23
				
					0
					 × 1

Write your answer

975 Views

7 Answers

3 years ago

one year ago