oh okay in that case i can use the output_ur params in uploading files methodl!!
Data in minio should be treat as an S3 bucket
CostlyOstrich36 could you explain the workflow with MinIO server setup which you just described? Should the data stored in MinIO be treated as a local folder setup and then maybe use this method! https://clear.ml/docs/latest/docs/clearml_data/clearml_data_sdk#syncing-local-storage
No problem. Sounds like a good solution, no need to implement something that has already been implemented somewhere else 🙂
A workaround can be to set up a local Minio server or upload to s3 directly, this way there shouldn't be a limit
BitterLeopard33 , ReassuredTiger98 , my bad. I just dug a bit in slack history, I think I got the issue mixed up with long file names 😞
Regarding http/chunking issue/solution - I can't find anything either. Maybe open a github issue / github feature request (for chunking files)
BitterLeopard33 , I don't see anything in GitHub. Maybe open an issue for better visibility ? 🙂
CostlyOstrich36 Thanks for the comment. Is there an Issue already opened and is tracking the status?
ReassuredTiger98 , BitterLeopard33 , I think I've encountered this 4GB http limit before. I think this should be fixed in next SDK release 🙂
I have a related question: I read here that 4GB is a http limitation and ClearML will not chunk single files. I take from that, that ClearML did not want/there was no need to implement an own solution so far. But what about models that are larger than 4GB?
WebApp: 1.2.0-153 • Server: 1.2.0-153 • API: 2.16
yes its a single file from local machine to a remote one.
BitterLeopard33 , Hi 🙂
What are you trying to upload? Is it a single file? What is the destination?