thanks, next time I will provide you will all the logs
And of course the full log for the experiment
This will print any server endpoint accessed by Trains during the experiment execution to the console
Well, it's a bit tricky, but here's what you'll need to configure. Basically we'd like to see what requests are being sent to the server, and the best way to do that is using the urllib3 log used by Python's requests module. Since this log is "muted" by default (to prevent it from littering the experiment's log), you'll need to add the following in your trains.conf file:sdk.log.disable_urllib3_info: false logging.loggers.urllib3 { level: DEBUG handlers: [urllib_console] } logging.handlers.urllib_console { class: logging.StreamHandler level: DEBUG }
Hi HealthyStarfish45 , how long would Task.init() hang for?
So a long time... I would start by getting all the docker containers logs
specifically the log for trains-apiserver
and the experiment did not produce any logs, shall I enable some debug flag?
apiserver logs were clean, only 200s there
By experiment log I actually meant the console output you have locally when running the experiment, not what is showing in the UI - from your description it's pretty obvious there's some issue preventing Trains from communicating property with the server 🙂
yes, but the local output was completely empty