SmugSnake6 I think the latest version (1.8.0) tries to parallelize it
You can also control max_workers

With default settings, to upload 2 datasets of 120 GB and 70 Gb it took more than 6 hours! And this is to upload the dataset on the server itself, the upload pipeline is done on the same computer as clearml

Sooo for the SHA2 generation, I've tested 2 very different CPUs, and it makes a HUGE difference 😅 I probably have to upgrade my server

With default settings, to upload 2 datasets of 120 GB and 70 Gb it took more than 6 hours!

SmugSnake6 at the end s the an outcome of limited bandwidth or limited CPU ?

From what I could see, generating SHA2:
i7-10700K: ~ 10 - 15 minutes Xeon E3-1240: 4 - 5 hours!Then in both cases I still have about an 1h30 to upload the images to the fileserver. Which I also find quite a bit slow, but the ClearML fileserver is on my old Xeon. I plan to upgrade my server and to test it again

Xeon E3-1240: 4 - 5 hours!wow... yes definitely worth upgrading 🙂