Is this just the console output while training?
There is clearly some connection to the ClearML server as it remains "running" the entire training session but there are no metrics or debug samples. And I see nothing in the logs to indicate there is an issue
The same training works sometimes. But I'm not sure how to troubleshoot when it stops logging the metrics
What happens if you're running the reporting example from the ClearML github repository?
I'm not sure how to even troubleshoot this.
Yes it shows on the UI and has the first epoch for some of the metrics but that's it. It has run like 50 epochs, it says it is still running but there are no updates to the scalars or debug samples
Hi @<1719524641879363584:profile|ThankfulClams64> , does the experiment itself show on the ClearML UI?