Likely related to: https://github.com/allegroai/clearml/issues/321 - clearML waits indefinitely on logs from TF v1

It seems there is some async behavior going on. After ending a run, this prompt just hangs for a long time:

2021-04-18 22:55:06,467 - clearml.Task - INFO - Waiting to finish uploads

And there's no sign of updates on the dashboard.

Still having the same issue I'm afraid

I guess I could do a backup of the DB and flush the data, but what I’m looking for is more of a “Select X experiments -> Send to blob storage” to free up space.

Got it to work now. Suspect something to do with a hydra issue, as both clearML hosted nor self-hosted option presented the same issue. Will update as I get a chance.

Can I force the login window?

Hi AgitatedDove14 , thanks for your reply.

I basically follow the example to use the SummaryWriter to add plots to the dashboard, however, the plots added are Plotly plots and not the tensorboard equivalent.

AgitatedDove14 , after some more pronging, the error seem to stem from the clearML server. The upload from client side does not seem to occur, or the server is not registering the uploads.

I've attempted to restart the server and pull the latest image, same error.

I might add, this error only started to occur upon upgrading from trains to clearmL

Scalars plot fine, images do not find their way to the dashboard

I see, no i do not get anything. I either get pop-ups when trying to capture the mpl figure, and otherwise nothing happens when using report_image .

I should mention this is run within a TF v1 session context

Yes, I am aware of this. None of the values are being reported, neither the scalar, images in debug-samples, nor in the plots tab.

I am able to register that the image exists, however, the push towards the clearML server just does not happen

Ahh excellent! Thanks AgitatedDove14 🙂

according to https://plotly.com/python/v3/LaTeX/ plotly should support Latex in labels since 3.6.0

Looks like we’re up and running

Noted. Thanks SuccessfulKoala55 - I was aware of this, yet I didn’t think it would also compare in the comparison window - It does. My god, I love your product.

Hi AgitatedDove14

Turns out my double loop caused some issues.

for e in range(num_epochs): for phase in ['train','valid']: for batch in dataloader:

A lot of bad requests and connection refused

Hi Martin, thanks for the swift response.

Yes, the artifacts, as backing up the full database would not resolve the question of capacity. Unless I’m missing something

Hi AgitatedDove14 , I am also unable to archive the individual experiments in it.

It's not a major issue, but it would be nice to remove as some users may get confused.

And now it works

That’s embarassing

I just changed the login details prior the restarting of the server, could this be at fault? I changed the trains-user to a different one

You were absolutely right