Sorry, been away for a while!
I have no additional information, since it was a bug in my model that I have since eliminated...
Maybe it was just a matplotlib error and can be dropped for now. I'll let you know if it pops up again!
Basically when there are occasionally extreme values (i.e. most values fall in [0, 50] range, and one value suddenly falls in 50e+12 range), the plotting library (matplotlib or ClearML, unsure) hangs for a really long time
Based on where it hangs, I think this has to do with reporting the scatter plots.
Is the task still reporting to the server? The UI should show the last updated time
My code is done, it's hanging waiting for ClearML :E
Still happening with 1.4.1 SuccessfulKoala55
There were a few related fixes - can you try with v1.4.1rc0?
how long does it take the task to get there?
Thanks for investigating this SweetBadger76 !
UnevenDolphin73 this message by itself does not indicate something is stuck, are you sure your code prints anything?
What is the clearml version you're using?
hey UnevenDolphin73
did you manage to get rid of this issue ?
Not really, I've only been able to somewhat understand the scope of where it happens, and I'm not sure it's even a ClearML issue (maybe matplotlib)
Trying now with 1.4.1, but I believe the changes you're referring to SuccessfulKoala55 were also introduced in 1.4.0, right?
Hey UnevenDolphin73
I have tried to reproduce the issue but with no success. I manage to auto report any graph designed according to your description - values between [0,50] and sudden extreme values. So far everything seems to be ok on my side. have you found something new reguarding this issue ? Could you send me more details on the graph which reporting hangs ?
Thanks
good to know. we will try to enquire that. thanks
Erm, one sec, let me find that thread...
Hm, this didn't happen until now; I'd be happy to try again with a new version, but something with 1.4.0 broke our StorageManager, so we reverted to 1.3.2