To expand on this, suppose I have an S3 bucket where my data is stored and I wish to transfer it to ClearML file server. I execute the following Python script

from clearml import Dataset

dataset = Dataset.create(dataset_name="my_dataset", dataset_project="my_project")

dataset.add_external_files(
  source_url="

", 
  dataset_path="/my_dataset/"
)
dataset.upload()
dataset.finalize()

and this is aws part of my clearml.conf

aws {
        s3 {
            # S3 credentials, used for read/write access by various SDK elements

            # The following settings will be used for any bucket not specified below in the "credentials" section
            # ---------------------------------------------------------------------------------------------------
            region: ""
            # Specify explicit keys
            key: "AKI***I5"
            secret: "2+1yd***2H6y"
            # Or enable credentials chain to let Boto3 pick the right credentials. 
            # This includes picking credentials from environment variables, 
            # credential file and IAM role using metadata service. 
            # Refer to the latest Boto3 docs
            use_credentials_chain: false
            # Additional ExtraArgs passed to boto3 when uploading files. Can also be set per-bucket under "credentials".
            extra_args: {}
            # ---------------------------------------------------------------------------------------------------


            credentials: [
                # specifies key/secret credentials to use when handling s3 urls (read or write)
                 {
                     bucket: "my_bucket"
                     key: "AKI***I5"
                     secret: "2+1yd***2H6y"
                 },
                # {
                #     # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
                #     host: "my-minio-host:9000"
                #     key: "12345678"
                #     secret: "12345678"
                #     multipart: false
                #     secure: false
                # }
            ]
        }

I noticed that while a dataset instance is generated on the ClearML dashboard, the data itself is not uploaded to the ClearML file server. I had assumed that this would be a straightforward process, apparently it’s not!