Unanswered
Hi.
I Have A Job That Processes Images And Creates ~5 Gb Of Processed Image Files (Lots Of Small Ones).
At The End - It Creates A
PanickyMoth78 there is no env var for sdk.google.storage.pool_connections/pool_maxsize
. We will likely add these env vars in a future release.
Yes, setting max_workers to 1 would not make a difference. The docs look a bit off, but it is specified that 1: if the upload destination is a cloud provider ('s3', 'gs', 'azure')
.
I'm thinking now that the memory issue might also be cause because of the fact that we prepare the zips in the background. Maybe a higher max_workers
would consume the zips faster. Might be counter intuitive, but I would try setting max_workers
to a higher number.
177 Views
0
Answers
one year ago
one year ago