Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Yo Guys, I'M Getting

yo guys,
I'm getting
Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f31b8561438>, 'Connection to OUR_IP timed out. (connect timeout=30.0)')'

when trying to log matplotlib image. everything else (logging models and metrics) works fine. any ideas?

  
  
Posted 4 years ago
Votes Newest

Answers 16


works like a charm! you guys are the best, as always =)

  
  
Posted 4 years ago

By the way, best practice for Trains when running on the cloud is to limit external access to 8081 to your office/home IP address

  
  
Posted 4 years ago

First thing to do - check the EC2 instance security group and make sure port 8081 is open

  
  
Posted 4 years ago

we're using EC2

  
  
Posted 4 years ago

yeah, that's exactly what I'm looking to right now 😃

  
  
Posted 4 years ago

The usual suspect is always a Firewall 🙂

  
  
Posted 4 years ago

Well, this suggests you have some network configuration issue that prevents requests from reaching the fileserver from your machine (and other machines, possible).
Is the server running on the same network, or is it hosted remotely (on AWS EC2, for example)?

  
  
Posted 4 years ago

everything is working as expected

  
  
Posted 4 years ago

OK, so first of all let's go over some basic stuff. I assume you're running Trains Server on some remote machine (i.e. not localhost), I also assume you can SSH to that machine... I also assume you're using docker-compose to run the server. In the server's console, can you try to:
Run sudo docker ps and see if the trains-fileserver docker container is up (and not restarting every few seconds) - its uptime should be roughly the same as that of the trains-apiserver or any of the other containers. Run curl http://localhost:8081 from the server's machine and see whether you get an error and not a timeout Open bash inside the trains-fileserver docker container using sudo docker exec -it trains-fileserver /bin/bash and try running curl http://localhost:8081 there, and see whether you get an error and not a timeout

  
  
Posted 4 years ago

no, we've only been using metric and model logging so far

  
  
Posted 4 years ago

DilapidatedDucks58 is this something that used to work (I mean with the exact same use of matplotlib) and is not working now? If so, was there any configuration change in Trains server or your network?

  
  
Posted 4 years ago

I get "The connection has timed out" when I'm trying to reach 8081 port

  
  
Posted 4 years ago

FYI matplotlib imshow will create a debug image, and on complex plots the plot might get converted to image. (But shown under the plots section). All in all you might not be aware of it, but you are uploading image to your files server

  
  
Posted 4 years ago

I'm asking since debug images are uploaded to the trains file server, and matplotlib is sometimes uploaded as an image as well. It might be there's a problem reaching the file server... Can you try browsing to http://YOURIP:8081/ ? You should reach the server but get a file not found error or something similar

  
  
Posted 4 years ago

nope

  
  
Posted 4 years ago

Hey DilapidatedDucks58 , are you logging debug images as well?

  
  
Posted 4 years ago