By the way, best practice for Trains when running on the cloud is to limit external access to 8081 to your office/home IP address
The usual suspect is always a Firewall 🙂
OK, so first of all let's go over some basic stuff. I assume you're running Trains Server on some remote machine (i.e. not localhost), I also assume you can SSH to that machine... I also assume you're using docker-compose to run the server. In the server's console, can you try to:
Run sudo docker ps and see if the trains-fileserver docker container is up (and not restarting every few seconds) - its uptime should be roughly the same as that of the trains-apiserver or any of the other containers. Run curl http://localhost:8081 from the server's machine and see whether you get an error and not a timeout Open bash inside the trains-fileserver docker container using sudo docker exec -it trains-fileserver /bin/bash and try running curl http://localhost:8081 there, and see whether you get an error and not a timeout
DilapidatedDucks58 is this something that used to work (I mean with the exact same use of matplotlib) and is not working now? If so, was there any configuration change in Trains server or your network?
Hey DilapidatedDucks58 , are you logging debug images as well?
I get "The connection has timed out" when I'm trying to reach 8081 port
no, we've only been using metric and model logging so far
works like a charm! you guys are the best, as always =)
I'm asking since debug images are uploaded to the trains file server, and matplotlib is sometimes uploaded as an image as well. It might be there's a problem reaching the file server... Can you try browsing to http://YOURIP:8081/ ? You should reach the server but get a file not found error or something similar
FYI matplotlib imshow will create a debug image, and on complex plots the plot might get converted to image. (But shown under the plots section). All in all you might not be aware of it, but you are uploading image to your files server
yeah, that's exactly what I'm looking to right now 😃
First thing to do - check the EC2 instance security group and make sure port 8081 is open
Well, this suggests you have some network configuration issue that prevents requests from reaching the fileserver from your machine (and other machines, possible).
Is the server running on the same network, or is it hosted remotely (on AWS EC2, for example)?