yes, but the local output was completely empty
apiserver logs were clean, only 200s there
thanks, next time I will provide you will all the logs
So a long time... I would start by getting all the docker containers logs
specifically the log for trains-apiserver
This will print any server endpoint accessed by Trains during the experiment execution to the console
And of course the full log for the experiment
Hi HealthyStarfish45 , how long would Task.init() hang for?
and the experiment did not produce any logs, shall I enable some debug flag?
By experiment log I actually meant the console output you have locally when running the experiment, not what is showing in the UI - from your description it's pretty obvious there's some issue preventing Trains from communicating property with the server 🙂
Well, it's a bit tricky, but here's what you'll need to configure. Basically we'd like to see what requests are being sent to the server, and the best way to do that is using the urllib3 log used by Python's requests module. Since this log is "muted" by default (to prevent it from littering the experiment's log), you'll need to add the following in your trains.conf file:sdk.log.disable_urllib3_info: false logging.loggers.urllib3 { level: DEBUG handlers: [urllib_console] } logging.handlers.urllib_console { class: logging.StreamHandler level: DEBUG }