Hi @<1742355077231808512:profile|DisturbedLizard6> , why would you like to upload from s3? You can simply register the s3 link. If you want to upload, the model path only supports local files
Based on this : it feels like S3 is supported
How are you using the function update_output_model
?
@<1523701087100473344:profile|SuccessfulKoala55> Hi, thank you. So, does ClearML have any direct interface to copy/upload files from S3 to its fileserver? Or do we need to download files locally first? I found InputModel.import_model
- is this the recommended way to import models from S3 to ClearML?
You are right. Based on this page S3 is supported. My file with this function:
from clearml import Task
task = Task.init(project_name='s3_upload_models', task_name='sklearn', output_uri=True)
task.update_output_model(auto_delete_file=False, name='v0.0.1', model_path='s3://<BUCKET_NAME>/MLOps/models/sklearn/sklearn.pkl')
task.close()
my clearml.conf
aws {
s3 {
# default, used for any bucket not specified below
key: ""
secret: ""
region: ""
credentials: [
{
# This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
host: "<s3host>:443"
key: "<ACCESS_KEY>"
secret: "<SECRET_KEY>"
multipart: false
secure: false
verify: false # OR "/path/to/ca/bundle.crt" OR "
" OR false to not verify
}
]
}
boto3 {
pool_connections: 512
max_multipart_concurrency: 16
multipart_threshold: 8388608 # 8MB
multipart_chunksize: 8388608 # 8MB
}
}