Hey all. Thinking about organisation of artefacts and how we store them in S3. We have been using the StorageManager to upload files to target S3 destinations. The issue with this approach is that you need to export the file to disk prior to the upload. Is there another way of exporting artefacts to target destinations which can do this without the prior export to disk?

Thinking to have a structure like:
raw data: <bucket>/raw-data/<partner>/<date>/
training/transformed data: <bucket>/training-data/<partner>/<date>/
models : <bucket>/models/<partner>/<date>/
data transformers: <bucket>/transformers/<partner>/<date>/
training-runs: <bucket>/training-runs/<partner>/<date>/

Another thing is that we have metadata exported for each artefact e.g. for raw data, you have:
partner name min timestamps in data max timestamp in data s3 location dataset ID task ID created by etc
The plan is to load the metadata exports into a separate s3 path to be loaded into the data warehouse so that we can query for analytics & deployment purposes. Any suggestions/insights on this?