So, Here'S A Question. Does Clearml Automatically Save Everything Necessary To Continue Training A Pytorch Language Model? Specifically, I'Ve Been Looking At The Checkpoint Folders Created When I'M Training A Huggingface Robertaformaskedlm. I Checked What

Unanswered

OK, I added

Task.current_task().upload_artifact(name='trainer_state', artifact_object=os.path.join(output_dir, "trainer_state.json"))

after this line:

https://github.com/huggingface/transformers/blob/040283170cd559b59b8eb37fe9fe8e99ff7edcbc/src/transformers/trainer.py#L1531

And it seems to be working.

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SmallDeer34
				
					0
					 × 1

97 Views

0 Answers

2 years ago

one year ago