hi, I created a dataset with 20K files, total of 20GB, with storage pointing to S3.
When I upload (or close) the dataset, during the compression phase, the clear-ml cli is killed due to OOM.Compressing file1.mp4 Compressing file2.mp4 Compressing file3.mp4 . . . KilledThe machine has 16 GB memory, I could always define swap memory but crashing seems excessive
Hi RoughTiger69 , how much ram is the process taking?
RAM=16
Task consumed 32GB memory total (had to add 16GB of swap)
What if you reduce the chunk size using --chunk-size ? Does it affect it?
I can try, but it will then damage the download speeds. Anyhow not a reasonable behavior in my opinion
