Hi PompousBeetle71
Try this one, let me know if it helpedlogging.getLogger('trains.frameworks').setLevel(ERROR)
PompousBeetle71 oh no 😞
okay this is a bit drastic, but let's see if it helps.
In your trains.conf, add the following section:loggers { loggers { trains { level: ERROR } } }
AgitatedDove14 Drastic indeed, I belive I will lose all the trains logs that way. In that case I prefer to keep the redundant logs.
If you'll find a more specific solution I'll love to know what it is 🙂
AgitatedDove14 I've tried the drastic measure suggested above as I had a log file of 1gb filled with the trains.frameworks - WARNING - Could not retrieve model location, skipping auto model logging
It didn't work :S
I actually tried to print the logging.getLogger("trains.frameworks").level
and it was ERROR as expected. Therefore I'm not quite sure that's the problem... next I thought to patch your functions.
the solution that worked: [logging.getLogger(name).setLevel(logging.ERROR) for name in logging.root.manager.loggerDict if "trains" in name]
PompousBeetle71 kudos on the solution!
What were the loggers you ended up setting?
I'd like to make sure we fix this issue
Also, is there a way to reproduce this issue of not capturing the model?
AgitatedDove14
These were the loggers names I can see locally running the code, it might differ running remotely.
['trains.utilities.pyhocon.config_parser', 'trains.utilities.pyhocon', 'trains.utilities', 'trains', 'trains.config', 'trains.storage', 'trains.metrics', 'trains.Repository Detection']
regarding repreduce it, have a long data processing after initializing the task and before setting the input model/output model.
Thanks PompousBeetle71
Quick question, what frameworks are you using?
Do you use save
method directly to file stream (or any other direct storage)?
I use torch and yes, I use save so your code will catch it.
PompousBeetle71 , the reason I'm asking is the warning you see is due to the fact it cannot detect the filename you are saving your model to ... I'm trying to figure out how that actually happened .
BTW: in the next version we will probably remove this warning altogether, but I'm still curious on how to reproduce 🙂
Hi PompousBeetle71
Could you test the latest RC, I think the warning were fixed:pip install trains==0.16.2rc0
Let me know...