Hi @<1742355077231808512:profile|DisturbedLizard6> , why would you like to upload from s3? You can simply register the s3 link. If you want to upload, the model path only supports local files
How are you using the function update_output_model
?
@<1523701087100473344:profile|SuccessfulKoala55> Hi, thank you. So, does ClearML have any direct interface to copy/upload files from S3 to its fileserver? Or do we need to download files locally first? I found InputModel.import_model
- is this the recommended way to import models from S3 to ClearML?
Based on this : it feels like S3 is supported
You are right. Based on this page S3 is supported. My file with this function:
from clearml import Task
task = Task.init(project_name='s3_upload_models', task_name='sklearn', output_uri=True)
task.update_output_model(auto_delete_file=False, name='v0.0.1', model_path='s3://<BUCKET_NAME>/MLOps/models/sklearn/sklearn.pkl')
task.close()
my clearml.conf
aws {
s3 {
# default, used for any bucket not specified below
key: ""
secret: ""
region: ""
credentials: [
{
# This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
host: "<s3host>:443"
key: "<ACCESS_KEY>"
secret: "<SECRET_KEY>"
multipart: false
secure: false
verify: false # OR "/path/to/ca/bundle.crt" OR "
" OR false to not verify
}
]
}
boto3 {
pool_connections: 512
max_multipart_concurrency: 16
multipart_threshold: 8388608 # 8MB
multipart_chunksize: 8388608 # 8MB
}
}