I looked at the values myself just in case. Tensorboard is right for sure 🙂

Here're the events.scalar_metrics payload:

{"meta":{"id":"08de8b5099dd4484a0aad19af2a140c2","trx":"08de8b5099dd4484a0aad19af2a140c2","endpoint":{"name":"events.scalar_metrics_iter_histogram","requested_version":"2.9","actual_version":"1.0"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":""},"data":{}}

Where does debug samples being stored? Maybe I can validate if they were uploaded to the server, I have an access to it

No, nothing

And events.get_task_plots

Here's the payload:

{"meta":{"id":"0fbbcd4fd5cf4b40a47997396521bae6","trx":"0fbbcd4fd5cf4b40a47997396521bae6","endpoint":{"name":"events.get_task_metrics","requested_version":"2.9","actual_version":"1.0"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":""},"data":{"metrics":[{"task":"9757604caf0342b6907d5eb672fe53cd","metrics":[]}]}}

I'll remove sensitive info and send it here

Yes, looks like it

Yes, there's a lot of DataCloneError:
` DataCloneError: The object could not be cloned. ExtensionChild.jsm:813
sendRemoveListener on closed conduit simple-translate@sienori.2199023255865 ConduitsChild.jsm:108
InvalidStateError: An attempt was made to use an object that is not, or is no longer, usable PictureInPictureChild.jsm:143
sendRemoveListener on closed conduit simple-translate@sienori.2199023255875 ConduitsChild.jsm:108
DataCloneError: The object could not be cloned. ExtensionChild.jsm:...

Ok guys we fixed it, the issue was an unavailable files_server. After I've changed iptables rules everything works fine

Also projects.get_all_ex method seems to be blocked

Also event.get_all_ex

No error, no warnings, images just don't show up

Thanks!

I see, the first rule of OSS projects. You need it - you do it 😄

I'll make a request then

What made you think it's wrong btw?

That's what I'm thinking, but I'm not sure what could be wrong

What are the relevant parts?

What does it look like?

AgitatedDove14 do you have any ETA on that feature? Can I do anything to speed up the devopment?

I see events.get_task_metrics call

No, it's like nothing happens at all

Yeah, that's it!

Another thing by the way: seems like when I try to upload images scalars cease to update too

Thanks AgitatedDove14 , already figured it out, but trains removed all data from mounted directory during repo checkout, so I ended up switching to absolute paths in my trainloop