WebApp: 1.2.0-153 • Server: 1.2.0-153 • API: 2.16
yes its a single file from local machine to a remote one.
No problem. Sounds like a good solution, no need to implement something that has already been implemented somewhere else 🙂
BitterLeopard33 , Hi 🙂
What are you trying to upload? Is it a single file? What is the destination?
I have a related question: I read here that 4GB is a http limitation and ClearML will not chunk single files. I take from that, that ClearML did not want/there was no need to implement an own solution so far. But what about models that are larger than 4GB?
ReassuredTiger98 , BitterLeopard33 , I think I've encountered this 4GB http limit before. I think this should be fixed in next SDK release 🙂
CostlyOstrich36 Thanks for the comment. Is there an Issue already opened and is tracking the status?
BitterLeopard33 , I don't see anything in GitHub. Maybe open an issue for better visibility ? 🙂
CostlyOstrich36 could you explain the workflow with MinIO server setup which you just described? Should the data stored in MinIO be treated as a local folder setup and then maybe use this method! https://clear.ml/docs/latest/docs/clearml_data/clearml_data_sdk#syncing-local-storage
oh okay in that case i can use the output_ur params in uploading files methodl!!
Data in minio should be treat as an S3 bucket
A workaround can be to set up a local Minio server or upload to s3 directly, this way there shouldn't be a limit
BitterLeopard33 , ReassuredTiger98 , my bad. I just dug a bit in slack history, I think I got the issue mixed up with long file names 😞
Regarding http/chunking issue/solution - I can't find anything either. Maybe open a github issue / github feature request (for chunking files)