• Thank you, I will give it a try. I debugged the code to investigate the cause of the failure. It appears that the code fails at line 1312 inside Dataset.create during the execution of the instance._serialize() function. I will further explore the code to identify the precise point of failure.

Hi @<1571308010351890432:profile|HurtAnt92> ! Yes, you can create intermediate datasets. Just batch your datasets, for each batch create new child datasets, then create a dataset that has as parents all of these resulting children.
I'm surprized you get OOM tho, we don't load the files in memory, just the name/path of the files + size, hash etc. Could there be some other factor that causes this issue?