Hi. Yup the model was not physically uploaded with the up:port into the bucket, although ClearML does indicate that it's there, except that I can't download it. I also verified this with another S3 client, the model was not there as well.
Can you not see the files using the ECS files browser?
Hi SubstantialElk6 , I have a few questions:
Are you running minio or some other service to store the data? Is http://ecs.ai the url of the server and clearml-models the bucket? Can you check if accessing by ip:port instead of url works?
The port (:80 in your case) is required, since otherwise both SDK and UI assume this is an AWS S3 bucket
Hi,
I'm running on Dell ECS storage appliance, which offers S3 compatibility.
yes http://ECS.ai is the DNS name of the server.
ClearML-models is the bucket.
Let me try with ip:port.
No, i can't see the files. But i can see if i don't use ':port' in the URL when uploading. I can't access the machine today, i'll try to check the S3 logs when i'm back.
SubstantialElk6 - Not sure I understood. When you set output_uri=' s3://ecs.ai:80/clearml-models/artifact ' , is the model not uploaded to the correct bucket?
I'm asking since as far as I know there was no change in the WebApp S3 driver from previous versions...
This is strange then, is it possible for clearml logs to register successfully saving into a S3 storage when actually it isn't? For example, i've seen in past experiences with certain S3 client that saved onto a local folder called 's3:/' instead of putting it on S3 storage itself.
Hi SubstantialElk6 , was this behaviour different in previous versions? If so, in which version?
Hi SubstantialElk6 ,
Can you add a screenshot of it? what do you have as MODEL URL ?
Well, a big is always possible, but we haven't seen that kind of behavior reported. As for storing in another place, that's only if the prefix somehow starts with a slash, and we know it doesn't since you can see that in the registered url.
SubstantialElk6 - can you please check the S3 configuration in your clearml.conf? Make sure if has the following:host: "ecs.ai:80" key: "your key" secret: "your secret" multipart: false secure: false
Hi TimelyPenguin76 ,
If you notice in the last screenshot, it state the bucket name to be http://ecs.ai . It then it tries to open http://s3.amazonaws.com/ecs.ai/clearml-models/artifact/uploading_file?X-Amz-Algorithm= ....
I didn't track the version on this change in behaviour. But last I tried it was able to download the content after I provide the credentials.
Hi, when i tried ip:port, it references the right host and bucket....BUT... the file is not found on the ECS S3 even though i can see from the logs that it states Completed model upload to s3://ecs.ai:80/clearml-models/artifacts/ ...