Out of curiosity, if Task flush worked, when did you get the error, at the end of the process ?
Yes makes sense, it sounded like that from the start. Luckily, the task.flush(...) way seems to work for now 🙂
Hi ScantChimpanzee51
btw: this seems like an S3 internal error
https://github.com/boto/s3transfer/issues/197
Ok so actually if I run task.flush(wait_for_uploads=True) at the end of the script it works ✔
So without the flush I got the error apparently at the very end of the script -
Yes... it's a python thing, background threads might get killed in random order, so that when one needs a background thread that died you get this error, which basically should mean you need to do the work in the calling thread.
This actually explains why calling Flush solved the issue.
Nice!
So without the flush I got the error apparently at the very end of the script - all commands of my actual Python code had run.