Unanswered
Hi,
I'M Experiencing Some Fairly Slow Uploads Of A New Dataset Version. I'M Running A Local Server And I'M Uploading A ~20Gb Update To A ~30Gb Dataset Consisting Of Few Hundreds Files, Each Up To Several Hundred Mbs. It Seems That Compressing And Upload I
Huh. So it looks like this was an issue of spawning too many upload workers which overwhelmed the fileserver limited to a single core...? When I limited max_workers in upload() on the client side, it went smoothly with no hanging. Funny thing is I had no issues with this using sync_folder() which I used for the original data upload, hence my perceived difference in performance despite similar file sizes.
134 Views
0
Answers
one year ago
one year ago