Is this a bug? Clearml version 1.26.4
im passing a list of S3 urls to add_external_files
The list is about 1K of files, but only 7 are uploaded (only unique file basenames)
Here is an example:
external_files =
[
' s3://s3.our-host.com:443/bucket/data/ a /layout_with_depth.jpg ',
' s3://s3.our-host.com:443/bucket/data/ b /layout_with_depth.jpg ',
' s3://s3.our-host.com:443/bucket/data/ c /layout_with_depth.jpg ',
' s3://s3.our-host.com:443/bucket/data/ a /file.jpg ',
' s3://s3.our-host.com:443/bucket/data/ b /file.jpg ',
' s3://s3.our-host.com:443/bucket/data/ c /file.jpg ',
]
clearml_dataset.add_external_files(
external_files
)
It only uploads:
' s3://s3.our-host.com:443/bucket/data/ a /layout_with_depth.jpg '
' s3://s3.our-host.com:443/bucket/data/ a /file.jpg '
We traced the issue to dataset.py (see attached image)