Thanks!

Thanks AgitatedDove14 , already figured it out, but trains removed all data from mounted directory during repo checkout, so I ended up switching to absolute paths in my trainloop

I looked at the values myself just in case. Tensorboard is right for sure 🙂

What made you think it's wrong btw?

Yes, looks like it

Another thing by the way: seems like when I try to upload images scalars cease to update too

No error, no warnings, images just don't show up

I'll remove sensitive info and send it here

Yes, there's a lot of DataCloneError:
` DataCloneError: The object could not be cloned. ExtensionChild.jsm:813
sendRemoveListener on closed conduit simple-translate@sienori.2199023255865 ConduitsChild.jsm:108
InvalidStateError: An attempt was made to use an object that is not, or is no longer, usable PictureInPictureChild.jsm:143
sendRemoveListener on closed conduit simple-translate@sienori.2199023255875 ConduitsChild.jsm:108
DataCloneError: The object could not be cloned. ExtensionChild.jsm:...

Here's the payload:

{"meta":{"id":"0fbbcd4fd5cf4b40a47997396521bae6","trx":"0fbbcd4fd5cf4b40a47997396521bae6","endpoint":{"name":"events.get_task_metrics","requested_version":"2.9","actual_version":"1.0"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":""},"data":{"metrics":[{"task":"9757604caf0342b6907d5eb672fe53cd","metrics":[]}]}}

Also projects.get_all_ex method seems to be blocked

What does it look like?

No, nothing

Yeah, that's it!

Ok guys we fixed it, the issue was an unavailable files_server. After I've changed iptables rules everything works fine

And events.get_task_plots

Here're the events.scalar_metrics payload:

{"meta":{"id":"08de8b5099dd4484a0aad19af2a140c2","trx":"08de8b5099dd4484a0aad19af2a140c2","endpoint":{"name":"events.scalar_metrics_iter_histogram","requested_version":"2.9","actual_version":"1.0"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":""},"data":{}}

No, it's like nothing happens at all

Where does debug samples being stored? Maybe I can validate if they were uploaded to the server, I have an access to it

I see events.get_task_metrics call

What are the relevant parts?

I see, the first rule of OSS projects. You need it - you do it 😄

AgitatedDove14 do you have any ETA on that feature? Can I do anything to speed up the devopment?

That's what I'm thinking, but I'm not sure what could be wrong

I'll make a request then

Also event.get_all_ex