Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Guys! Can Clearml Upload Models From S3 To A File Server? I Tried To Use The 'Update_Output_Model' Function And Changed Clearml.Conf, But In The End It Gave Me An Error: "Exception Encountered While Uploading [Errno 2] No Such File Or Directory: 'S3://

Hi guys! Can ClearML upload models from S3 to a file server? I tried to use the 'update_output_model' function and changed clearml.conf, but in the end it gave me an error: "Exception encountered while uploading [Errno 2] No such file or directory: 's3://....'" Or maybe ClearML doesn't have this functionality?

  
  
Posted 22 hours ago
Votes Newest

Answers 4


You are right. Based on this page S3 is supported. My file with this function:

from clearml import Task

task = Task.init(project_name='s3_upload_models', task_name='sklearn', output_uri=True)
task.update_output_model(auto_delete_file=False, name='v0.0.1', model_path='s3://<BUCKET_NAME>/MLOps/models/sklearn/sklearn.pkl')
task.close()

my clearml.conf

aws {
    s3 {
        # default, used for any bucket not specified below
        key: ""
        secret: ""
        region: ""

        credentials: [
            {
                # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
                host: "<s3host>:443"
                key: "<ACCESS_KEY>"
                secret: "<SECRET_KEY>"
                multipart: false
                secure: false
                verify: false # OR "/path/to/ca/bundle.crt" OR "
" OR false to not verify
            }
        ]
    }


    boto3 {
        pool_connections: 512
        max_multipart_concurrency: 16
        multipart_threshold: 8388608 # 8MB
        multipart_chunksize: 8388608 # 8MB
    }
}
  
  
Posted 20 hours ago

Based on this : it feels like S3 is supported

  
  
Posted 21 hours ago

How are you using the function update_output_model ?

  
  
Posted 21 hours ago

Hi @<1742355077231808512:profile|DisturbedLizard6> , why would you like to upload from s3? You can simply register the s3 link. If you want to upload, the model path only supports local files

  
  
Posted 11 hours ago