Unanswered
Hi. I Spent Some Time This Week Trying To Optimise File Transfer Time In And Out Of Processes That Use Google'S Gcs (In Vertex Ai Pipelines).
It Seems That In The Case Where I Have A Lot Of Very Small Files, It Made More Sense To Tar.Gz Them And Send A Bi
Generally speaking, for the exact reason if you are passing a list of files, or a folder, it will actually zip them and upload the zip file. Specifically to pipeline it should be similar. BTW I think you can change the number of parallel upload threads in StorageManager, but as you mentioned it is faster to zip into one file. Make sense?
177 Views
0
Answers
2 years ago
one year ago