Reputation
Badges 1
50 × Eureka!I am able to register that the image exists, however, the push towards the clearML server just does not happen
Thanks for the rapid reply
I just changed the login details prior the restarting of the server, could this be at fault? I changed the trains-user to a different one
Can I force the login window?
I'll see if I can get time to do a reproducible example.
The issue only arises upon sending Images. (Both numpy, mpl and PIL)
AgitatedDove14 , after some more pronging, the error seem to stem from the clearML server. The upload from client side does not seem to occur, or the server is not registering the uploads.
I've attempted to restart the server and pull the latest image, same error.
Hang on, It was the static - No auto update AMI:ami-0d6f44a1a7145a9f8
Yes, I am aware of this. None of the values are being reported, neither the scalar, images in debug-samples, nor in the plots tab.
I'm running in the command line with Hydra and TFv1
I'm afraid I just had an improper indentation somewhere, no harms done. Thanks for the reply.
No, but I am able to delete other folder with experiements
However in this case the option is grayed out
Hi Martin, thanks for the swift response.
Yes, the artifacts, as backing up the full database would not resolve the question of capacity. Unless I’m missing something
Looks like we’re up and running
Still having the same issue I'm afraid
Thank you 😅 Very much
Hi SuccessfulKoala55 , I believe I was only given one option in my region (EU Stockholm) which was the 0.16.1 version with the AMI location:
aws-marketplace/allegroai-trains-server-0.16.1-320-273-c5c210e4-5094-4eb9-a613-a32c0378de31-ami-06f5e9f4dfa499dca.4
I used the Trains AMI, and I am not sure whether it was the auto-updated or static one
Ahh excellent! Thanks AgitatedDove14 🙂
Not sure if you've tried this, but would click work? Otherwise the simplest thing I know is possible in Click, is to send the Dict of arguments further down the pipeline for the Allegro callback listener.
A lot of bad requests and connection refused
Additionally -
Are there any clever functionality for dumping experiment data to external storage to avoid filling up the server?
It seems there is some async behavior going on. After ending a run, this prompt just hangs for a long time:
2021-04-18 22:55:06,467 - clearml.Task - INFO - Waiting to finish uploads
And there's no sign of updates on the dashboard.
And another detail, upon running the same code in a notebook session, everything gets stored as intended (to clearML dashboard)
Upon removing the phase loop the epoch was detected automatically again.
Noted. Thanks SuccessfulKoala55 - I was aware of this, yet I didn’t think it would also compare in the comparison window - It does. My god, I love your product.
Yes, the login details to the Trains UI

