im also batch uploading, maybe thats the problem?

• The dataset is about 1TB containing 1 million files
• I dont have the SSD space locally to do the upload
• So i download a part of the dataset, use add_files() and then upload() to that batch
• Upload the dataset
  I noticed that each batch is slower and slower

No, i specify where to upload
I see the data on S3 bucket is beeing uploaded. Just the log messages are really confusing

@<1590514584836378624:profile|AmiableSeaturtle81> note that we zip the files before uploading them as artifacts to the dataset task. Any chance you are specifying the default output uri as being a local path, such as /tmp ?

Hi @<1590514584836378624:profile|AmiableSeaturtle81> ! What function are you using to upload the data?