@<1523701083040387072:profile|UnevenDolphin73> : I see. I did not make the connection that output_uri=True
is what I was missing. I thought this was the default. But the default is actually "None", which is different than "True".
Heh, well, John wrote that in the first reply in this thread ๐
And in Task.init
main documentation page (nowhere near the code), it says the following -
@<1523704157695905792:profile|VivaciousBadger56> It seems like whatever you pickled in the zip file relies on some additional files that are not pickled.
But, I guess @<1523701070390366208:profile|CostlyOstrich36> wrote that in a different chat, right?
The documentation is messy, Iโve complained about it the in the past too ๐
@<1523701070390366208:profile|CostlyOstrich36>
My training outputs a model as a zip file. The way I save and load the zip file to make up my model is custom made (no library is directly used), because we invented the entire modelling ourselves. What I did so far:
output_model = OutputModel(task=..., config_dict={...}, name=f"...")
output_model.update_weights("C:\io__path\...", is_package=True)
and I am trying to load the model in a different Python process with
mymodel = task.models['output'][0]
mymodel = mymodel.get_local_copy(extract_archive=True, raise_on_error=True)
and I get in the clearml cache a .
training.pt file, which seems to be some kind of archive. Inside I have two files named data.pkl
and version
and a folder with the two files named 86922176
and 86934640
.
I am not sure how to proceed after trying to use pickle, zip and joblib. I am kind of at a loss. I suspect, my original zip file might be somehow inside, but I am not sure.
Sure, we could simply use the generic artifacts sdk, but I would like to use the available terminological methods and functions.
How should I proceed?
We're certainly working hard on improving the documentation (and I do apologize for the frustrating experience)
Do you mean "exactly" as in "you finally got it" or in the sense of "yes, that was easy to miss"?
Yes, you're correct, I misread the exception.
Maybe it hasn't completed uploading? At least for Datasets one needs to explicitly wait IIRC
@<1523701083040387072:profile|UnevenDolphin73> : How do you figure? In the past, my colleagues and I just shared the .zip
file via email / MS Teams and it worked. So I don't think so.
@<1523701083040387072:profile|UnevenDolphin73>
@<1523701087100473344:profile|SuccessfulKoala55> Also, I think that - in this case, but also in other cases - the issue is not just the documentation, but also the design of the SDK.
@<1523701083040387072:profile|UnevenDolphin73> : Thanks, but it does not mention the File Storage of "ClearML Hosted Server".
We'll try to add referenced to that in other places as well ๐
Hi @<1523704157695905792:profile|VivaciousBadger56> , you can configure Task.init(..., output_uri=True)
and this will save the models to the clearml file server
@<1523701083040387072:profile|UnevenDolphin73> : I do not see any way to download the model manually from the web app either. All I see is the link to the file on my harddrive (see shreenshot).
The second process says there is not file at all. I think, all that happened is that the update_weights
only uploaded the location of the .zip
file (which we denote as a .model
file) on my harddrive, but not the file itself.
I have already been trying to contribute (have three pull requests), but honestly I feel it is a bit weird, that I need to update a documentation about something I do not understand, while I actually try to evaluate if ClearML is the right tool for our company...
Heh, good @<1523704157695905792:profile|VivaciousBadger56> ๐
I was just repeating what @<1523701070390366208:profile|CostlyOstrich36> suggested, credits to him
By the way, output_uri is also documented as part of the Task.init() docstring ( None )
Well you could start by setting the output_uri
to True
in Task.init
.
I can only say Iโve found ClearML to be very helpful, even given the documentation issue.
I think theyโve been working on upgrading it for a while, hopefully something new comes out soon.
Maybe @<1523701205467926528:profile|AgitatedDove14> has further info ๐
FWIW, we prefer to set it in the agentโs configuration file, then itโs all automatic
@<1523701083040387072:profile|UnevenDolphin73> : From which URL is your most recent screenshot?