yes, but the local output was completely empty
Hi HealthyStarfish45 , how long would Task.init()
hang for?
Well, it's a bit tricky, but here's what you'll need to configure. Basically we'd like to see what requests are being sent to the server, and the best way to do that is using the urllib3
log used by Python's requests
module. Since this log is "muted" by default (to prevent it from littering the experiment's log), you'll need to add the following in your trains.conf
file:sdk.log.disable_urllib3_info: false logging.loggers.urllib3 { level: DEBUG handlers: [urllib_console] } logging.handlers.urllib_console { class: logging.StreamHandler level: DEBUG }
This will print any server endpoint accessed by Trains during the experiment execution to the console
thanks, next time I will provide you will all the logs
So a long time... I would start by getting all the docker containers logs
and the experiment did not produce any logs, shall I enable some debug flag?
And of course the full log for the experiment
apiserver logs were clean, only 200s there
By experiment log I actually meant the console output you have locally when running the experiment, not what is showing in the UI - from your description it's pretty obvious there's some issue preventing Trains from communicating property with the server 🙂
specifically the log for trains-apiserver