Unanswered
Hello, For Some Reason My Upload Speed To S3 Is Insanely Slow, I Noticed In Logs That It Upoads To /Tmp Folder. What Does That Mean? Why Tmp?
im also batch uploading, maybe thats the problem?
- The dataset is about 1TB containing 1 million files
- I dont have the SSD space locally to do the upload
- So i download a part of the dataset, use add_files() and then upload() to that batch
- Upload the dataset
I noticed that each batch is slower and slower
69 Views
0
Answers
6 months ago
6 months ago