The use case is that we want a dataset of over 100k S3 images and they are scattered all over our bucket due to how we organise those images. If I send an array of URLs as the source_url it will eventually fail after around 40 due to the GCS rate limit for updating the state.json .