Have you ever benchmarked clear ml datasets on large datasets ? How good is it on handling them ?

It seems to like the process exits on a memory error

@<1523701087100473344:profile|SuccessfulKoala55> any hints ?

Nothing special

    dataset = Dataset.create(dataset_name = 'my_dataset', parent_datasets=None, use_current_task=False)
    dataset.add_files(dataset_dir, verbose=False)
    dataset.upload(output_url='

)
    dataset.finalize(verbose=True)

Hi @<1523702307240284160:profile|TeenyBeetle18> , how are you converting it, what code are you using exactly?