specifically the log for trains-apiserver
This will print any server endpoint accessed by Trains during the experiment execution to the console
By experiment log I actually meant the console output you have locally when running the experiment, not what is showing in the UI - from your description it's pretty obvious there's some issue preventing Trains from communicating property with the server 🙂
So a long time... I would start by getting all the docker containers logs
and the experiment did not produce any logs, shall I enable some debug flag?
yes, but the local output was completely empty
And of course the full log for the experiment
Well, it's a bit tricky, but here's what you'll need to configure. Basically we'd like to see what requests are being sent to the server, and the best way to do that is using the urllib3
log used by Python's requests
module. Since this log is "muted" by default (to prevent it from littering the experiment's log), you'll need to add the following in your trains.conf
file:sdk.log.disable_urllib3_info: false logging.loggers.urllib3 { level: DEBUG handlers: [urllib_console] } logging.handlers.urllib_console { class: logging.StreamHandler level: DEBUG }
Hi HealthyStarfish45 , how long would Task.init()
hang for?
thanks, next time I will provide you will all the logs
apiserver logs were clean, only 200s there