UnevenDolphin73 : From which URL is your most recent screenshot?
It should store it on the fileserver, perhaps you're missing a configuration option somewhere?
It is documented at None ... super deep in the code. If you don't know that output_uri in TASK's (!) init is relevant, you would never know...
Heh, well, John wrote that in the first reply in this thread 🙂
And in Task.init main documentation page (nowhere near the code), it says the following -
FWIW It’s also listed in other places VivaciousBadger56 , e.g. None says:
In order to make sure we also automatically upload the model snapshot (instead of saving its local path), we need to pass a storage location for the model files to be uploaded to.
For example, upload all snapshots to an S3 bucket…
Do you mean "exactly" as in "you finally got it" or in the sense of "yes, that was easy to miss"?
VivaciousBadger56 I'm not sure I'm following you - is the issue not being able to upload to the ClearML server or to load the downloaded file?
We'll try to add referenced to that in other places as well 👍
FWIW, we prefer to set it in the agent’s configuration file, then it’s all automatic
Yes, you're correct, I misread the exception.
Maybe it hasn't completed uploading? At least for Datasets one needs to explicitly wait IIRC
By the way, output_uri is also documented as part of the Task.init() docstring ( None )
UnevenDolphin73 : Thanks, but it does not mention the File Storage of "ClearML Hosted Server".
The documentation is messy, I’ve complained about it the in the past too 🙈
Hi all, sorry for not being so responsive today 🙏
I can only say I’ve found ClearML to be very helpful, even given the documentation issue.
I think they’ve been working on upgrading it for a while, hopefully something new comes out soon.
Maybe AgitatedDove14 has further info 🙂
I have already been trying to contribute (have three pull requests), but honestly I feel it is a bit weird, that I need to update a documentation about something I do not understand, while I actually try to evaluate if ClearML is the right tool for our company...
We're certainly working hard on improving the documentation (and I do apologize for the frustrating experience)
UnevenDolphin73 : I do not get this impression, because during update_weights I get the message
2023-02-21 13:54:49,185 - clearml.model - INFO - No output storage destination defined, registering local model C:\Users..._Demodaten_FF_2023-02-21_13-53-51.624362.model
SuccessfulKoala55 : I referenced this conversation in the issue None
Heh, good VivaciousBadger56 😁
I was just repeating what CostlyOstrich36 suggested, credits to him
UnevenDolphin73 : How do you figure? In the past, my colleagues and I just shared the .zip file via email / MS Teams and it worked. So I don't think so.
SuccessfulKoala55 Also, I think that - in this case, but also in other cases - the issue is not just the documentation, but also the design of the SDK.
SuccessfulKoala55 I think I might have made a mistake earlier - but not in the code I posted before. Now, I have the following situation:
- In my training Python process on my notebook I train the custom made model and put it on my harddrive as a zip file. Then I run the code
output_model = OutputModel(task=task, config_dict={...}, name=f"...")
output_model.update_weights(weights_filename=r"C:\path\to\mymodel.zip", is_package=True)
-
I delete the "C:\path\to\mymodel.zip", because it would not be available on my colleagues' computers.
-
In a second process, the model-inference process, I run
mymodel = task.models['output'][-1]
mymodel = mymodel.get_local_copy(extract_archive=True, raise_on_error=True)
and get the error
ValueError: Could not retrieve a local copy of model weights 8ad4db1561474c43b0747f7e69d241a6, failed downloading
I do not have an aws S3 instance or something like that. This is why I would like to store my mymodel.zip file directly on the ClearML Hosted Service. The model is around 2MB large.
How should I proceed?