Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Have A Local Version Of Trains Server (Installed On A Vm In Gcp), One Problem I Run Into Was That Task.Init() Would Hang, And In The Ui I Could See "Drafts" Of Experiments. After Doing Docker-Compose Down And Up, Everything Went Back To Normal. Unfortun

I have a local version of trains server (installed on a VM in GCP), one problem I run into was that Task.init() would hang, and in the UI I could see "drafts" of experiments. After doing docker-compose down and up, everything went back to normal. Unfortunately I did not inspect the logs carefuly enough before restarting the server, so I have no trace of what was not working. If it happens again in the future, what should I look into for debugging?

  
  
Posted 3 years ago
Votes Newest

Answers 15


Hi HealthyStarfish45 , how long would Task.init() hang for?

  
  
Posted 3 years ago

SuccessfulKoala55 20 minutes at least

  
  
Posted 3 years ago

By experiment log I actually meant the console output you have locally when running the experiment, not what is showing in the UI - from your description it's pretty obvious there's some issue preventing Trains from communicating property with the server 🙂

  
  
Posted 3 years ago

So a long time... I would start by getting all the docker containers logs

  
  
Posted 3 years ago

yes, but the local output was completely empty

  
  
Posted 3 years ago

specifically the log for trains-apiserver

  
  
Posted 3 years ago

apiserver logs were clean, only 200s there

  
  
Posted 3 years ago

Let me take a look

  
  
Posted 3 years ago

Well, it's a bit tricky, but here's what you'll need to configure. Basically we'd like to see what requests are being sent to the server, and the best way to do that is using the urllib3 log used by Python's requests module. Since this log is "muted" by default (to prevent it from littering the experiment's log), you'll need to add the following in your trains.conf file:
sdk.log.disable_urllib3_info: false logging.loggers.urllib3 { level: DEBUG handlers: [urllib_console] } logging.handlers.urllib_console { class: logging.StreamHandler level: DEBUG }

  
  
Posted 3 years ago

ok, thanks!

  
  
Posted 3 years ago

I see...

  
  
Posted 3 years ago

thanks, next time I will provide you will all the logs

  
  
Posted 3 years ago

This will print any server endpoint accessed by Trains during the experiment execution to the console

  
  
Posted 3 years ago

And of course the full log for the experiment

  
  
Posted 3 years ago

and the experiment did not produce any logs, shall I enable some debug flag?

  
  
Posted 3 years ago
569 Views
15 Answers
3 years ago
one year ago
Tags