I have a related question: I read here that 4GB is a http limitation and ClearML will not chunk single files. I take from that, that ClearML did not want/there was no need to implement an own solution so far. But what about models that are larger than 4GB?
CostlyOstrich36 Thanks for the comment. Is there an Issue already opened and is tracking the status?
CostlyOstrich36 could you explain the workflow with MinIO server setup which you just described? Should the data stored in MinIO be treated as a local folder setup and then maybe use this method! https://clear.ml/docs/latest/docs/clearml_data/clearml_data_sdk#syncing-local-storage
oh okay in that case i can use the output_ur params in uploading files methodl!!
WebApp: 1.2.0-153 • Server: 1.2.0-153 • API: 2.16
BitterLeopard33 , I don't see anything in GitHub. Maybe open an issue for better visibility ? 🙂
BitterLeopard33 , ReassuredTiger98 , my bad. I just dug a bit in slack history, I think I got the issue mixed up with long file names 😞
Regarding http/chunking issue/solution - I can't find anything either. Maybe open a github issue / github feature request (for chunking files)
BitterLeopard33 , Hi 🙂
What are you trying to upload? Is it a single file? What is the destination?
A workaround can be to set up a local Minio server or upload to s3 directly, this way there shouldn't be a limit
yes its a single file from local machine to a remote one.
ReassuredTiger98 , BitterLeopard33 , I think I've encountered this 4GB http limit before. I think this should be fixed in next SDK release 🙂
Data in minio should be treat as an S3 bucket
No problem. Sounds like a good solution, no need to implement something that has already been implemented somewhere else 🙂