Hi, Some Plots Don'T Render Via

Answered

Hi, some plots don't render via report_plotly during training past certain point, but when it's completed, they do render. If i abort, they're not rendered. plotly itself is rendering them fine in IPython.

I suspected there was some race condition with rendering something data-rich, put 30sec sleep timer in the training loop but it did not help

I've sanitized the input by removing inf/-inf and NaN from the plotted data( report_plotly struggles with this), it didnt help

What else can i try?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					GloriousPenguin2
				
					0
					 × 1

Votes Newest

Answers 5

Upd: cant reproduce "rendered on some iterations", now its only: not rendered until run is completed.

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					GloriousPenguin2
				
					0
					 × 1

But the issue still reproduces?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

What happens during the run is that plotly plots are shown during run on your computer but they don't show in UI and ONLY after the run is finished the plots show up?

Are your runs long?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Upd: fixed the error, it was in some unrelated code

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					GloriousPenguin2
				
					0
					 × 1

CostlyOstrich36

Updated.

They are rendered on some iterations, especially early iterations. Rest is correct

The testing runs are short but real runs will be long so its very undesirable

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					GloriousPenguin2
				
					0
					 × 1

Write your answer

960 Views

5 Answers

2 years ago

one year ago