Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Guys! Can Clearml Upload Models From S3 To A File Server? I Tried To Use The 'Update_Output_Model' Function And Changed Clearml.Conf, But In The End It Gave Me An Error: "Exception Encountered While Uploading [Errno 2] No Such File Or Directory: 'S3://

Hi guys! Can ClearML upload models from S3 to a file server? I tried to use the 'update_output_model' function and changed clearml.conf, but in the end it gave me an error: "Exception encountered while uploading [Errno 2] No such file or directory: 's3://....'" Or maybe ClearML doesn't have this functionality?

  
  
Posted one month ago
Votes Newest

Answers 5


Hi @<1742355077231808512:profile|DisturbedLizard6> , why would you like to upload from s3? You can simply register the s3 link. If you want to upload, the model path only supports local files

  
  
Posted one month ago

You are right. Based on this page S3 is supported. My file with this function:

from clearml import Task

task = Task.init(project_name='s3_upload_models', task_name='sklearn', output_uri=True)
task.update_output_model(auto_delete_file=False, name='v0.0.1', model_path='s3://<BUCKET_NAME>/MLOps/models/sklearn/sklearn.pkl')
task.close()

my clearml.conf

aws {
    s3 {
        # default, used for any bucket not specified below
        key: ""
        secret: ""
        region: ""

        credentials: [
            {
                # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
                host: "<s3host>:443"
                key: "<ACCESS_KEY>"
                secret: "<SECRET_KEY>"
                multipart: false
                secure: false
                verify: false # OR "/path/to/ca/bundle.crt" OR "
" OR false to not verify
            }
        ]
    }


    boto3 {
        pool_connections: 512
        max_multipart_concurrency: 16
        multipart_threshold: 8388608 # 8MB
        multipart_chunksize: 8388608 # 8MB
    }
}
  
  
Posted one month ago

Based on this : it feels like S3 is supported

  
  
Posted one month ago

@<1523701087100473344:profile|SuccessfulKoala55> Hi, thank you. So, does ClearML have any direct interface to copy/upload files from S3 to its fileserver? Or do we need to download files locally first? I found InputModel.import_model - is this the recommended way to import models from S3 to ClearML?

  
  
Posted one month ago

How are you using the function update_output_model ?

  
  
Posted one month ago