Hi RoughTiger69
unfortunately, the model was serialized with a different module structure - it was originally placed in a (root) module called
model
....
Is this like a pickle issue?
Unfortunately, this doesn’t work inside clear.ml since there is some mechanism that overrides the import mechanism using
import_bind
.
__patched_import3
What error are you getting? (meaning why isn't it working)
it is a pickle issue
‘package model doesn’t exist’
Sounds like it, why do you think clearml has anything there ?
BTW:
import_bind
.
__patched_import3
this is just so when packages that clearml autoconnects with are patched if imported After Task.init was called.
RoughTiger69 can you share the python version and the logs?
I think it has something to do with clearml since I can run this code as pure python without clearml, and when I activate clearml, I see that torch.load() hits the
import_bind
.
__patched_import3
when trying to deserialize the saved model
python 3.8
I’ve worked around the issue by doing:sys.modules['model'] = local_model_package
I will try and get back to this area of the code soon
it is a pickle issue
‘package model doesn’t exist’
despite me attempting to add the right path to sys.path right before loading
What's the python, torch, clearml version?
Any chance this can be reproducible ?
What's the full error trace/stack you are getting?
Can you try to debug it to where exactly it fails here?
https://github.com/allegroai/clearml/blob/86586fbf35d6bdfbf96b6ee3e0068eac3e6c0979/clearml/binding/import_bind.py#L48
RoughTiger69 wdyt?
Nice workaround!
RoughTiger69 how do I reproduce this behavior? (I'm still unsure on why exactly the clearml binding broke it, and would like to fix that)
(can you also provide the crash trace, maybe that could help as well)