Yes using clearml-data.
Can I pass a s3 path to ds.add_files() essentially so that I can directly store a dataset without having to get the files to local and then upload again. Makes sense?
Why not using it directly from S3?
You can https://allegro.ai/clearml/docs/docs/examples/examples_storagehelper.html#downloading-a-file it with the storageManager
When uploading the files, hash is being calculated for every entry, and this is done for the local files. so currently clearml-data support local files.
What would you like to do with the dataset? Why not using it directly from S3?
Hi TrickySheep9 , didnt get the idea, you are using clearml-data
? you just want to upload a local folder to S3?