Oh okay, so you’re saying the UI will always use plotly to display the data? I guess I got confused since the color choices in https://allegro.ai/docs/examples/frameworks/pytorch/pytorch_tensorboard/ made it look like the TensorBoard graphs.

Upon removing the phase loop the epoch was detected automatically again.

I'm afraid I just had an improper indentation somewhere, no harms done. Thanks for the reply.

Hi AgitatedDove14

Turns out my double loop caused some issues.

for e in range(num_epochs): for phase in ['train','valid']: for batch in dataloader:

It seems there is some async behavior going on. After ending a run, this prompt just hangs for a long time:

2021-04-18 22:55:06,467 - clearml.Task - INFO - Waiting to finish uploads

And there's no sign of updates on the dashboard.

Hi SuccessfulKoala55 , I believe I was only given one option in my region (EU Stockholm) which was the 0.16.1 version with the AMI location:

aws-marketplace/allegroai-trains-server-0.16.1-320-273-c5c210e4-5094-4eb9-a613-a32c0378de31-ami-06f5e9f4dfa499dca.4

I used the Trains AMI, and I am not sure whether it was the auto-updated or static one

Yes, exactly, from a previously executed run. Essentially, I write a grid of images which are supposed to learn a generative task, i’d like to download all images and generate a gif from the collection of images.

Still having the same issue I'm afraid

And now it works

Not sure if you've tried this, but would click work? Otherwise the simplest thing I know is possible in Click, is to send the Dict of arguments further down the pipeline for the Allegro callback listener.

Ahh excellent! Thanks AgitatedDove14 🙂

I guess I could do a backup of the DB and flush the data, but what I’m looking for is more of a “Select X experiments -> Send to blob storage” to free up space.

Hi Martin, thanks for the swift response.

Yes, the artifacts, as backing up the full database would not resolve the question of capacity. Unless I’m missing something

No, but I am able to delete other folder with experiements

Many thanks! Got to say; You’ve made and are making a killer of a product!

I just changed the login details prior the restarting of the server, could this be at fault? I changed the trains-user to a different one

Additionally -

Are there any clever functionality for dumping experiment data to external storage to avoid filling up the server?

Hi AgitatedDove14 - I used the Tensorboard writer.scalars function. Haven't tried the Plotly natively, but I guess its the same, since I imagine you're just doing a passthrough.

And another detail, upon running the same code in a notebook session, everything gets stored as intended (to clearML dashboard)

Thank you 😅 Very much

Please don’t ping people here.