So without the flush I got the error apparently at the very end of the script - all commands of my actual Python code had run.
Yes makes sense, it sounded like that from the start. Luckily, the task.flush(...) way seems to work for now 🙂
Out of curiosity, if Task flush worked, when did you get the error, at the end of the process ?
Hi ScantChimpanzee51
btw: this seems like an S3 internal error
https://github.com/boto/s3transfer/issues/197
So without the flush I got the error apparently at the very end of the script -
Yes... it's a python thing, background threads might get killed in random order, so that when one needs a background thread that died you get this error, which basically should mean you need to do the work in the calling thread.
This actually explains why calling Flush solved the issue.
Nice!
Ok so actually if I run task.flush(wait_for_uploads=True) at the end of the script it works ✔