Thank you. I am still having the issue. I verified that output_uri
of Task.init works and also clearml-data
with MinIO storage works, but the logger still throws errors
Is this really working for you guys? I have no clue what's wrong. Seems so unlikely that my code works with artifacts, datasets, but not logging...
Unfortunately, not. Quick question: Is there caching happening somewhere besides .clearml
? Does the boto3 driver create cache?
Okay, but are you logs still stored on MinIO with only using sdk.development.default_output_uri
?
The logs are on the bucket, yes.
The default file server is also set to s3://ip:9000/clearml
It is. Let me see what else I have set up for MinIO in configs, one moment
Hey ReassuredTiger98
Is there any update from your side ?
I confirm that you need to put your key and secret in the credentials section of the configuration file . As Idan, I let my policy configuration untouched
Also I can see that clearml correctly loads the configSTORAGE S3BucketConfig(bucket='clearml', host='myhost:9000', key='mykey' secret='mysecret', token='', multipart=False, acl='', secure=True, region=None, verify=True, use_credentials_chain=False)
oups yes, you are right. output_uri is used for the artifacts
for the logger it is https://clear.ml/docs/latest/docs/references/sdk/logger#set_default_upload_destination
btw what do you get when you do task.get_logger().get_default_upload_destination() ?
This is the error I get from setting the logger upload destination.botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.
But this means the logger will use the default fileserver or not?
as long as you dont precise any uri when you init a task (with default outuput uri parameter), clearml will use the config file value registered into sdk.development.default_output_uri
The api.files_server
is set to the MinIO endpoint s3://ip:9000/clearml (both locally and remotely) The sdk.development.default_output_uri
is set to the MinIO endpoint (both locally and remotely) When we call Task.init
I do not set the output_uri
at all I get the logger directly with task.get_logger()
Is there a simple way to get the response of the MinIO instance? Then I can verify whether it is the MinIO instance or my client
Is sdk.development.default_output_uri
used with s3://ip:9000/clearml or
ip:9000/clearml
?
After setting the sdk.development.default_output_uri
in the configs, my code kinda looks like:
` task = Task.init(project_name=..., task_name=..., tags=...)
logger = task.get_logger()
report with logger freely `
mytask.get_logger().current_logger().set_default_upload_destination("
s3://ip:9000/clearml ")
this is what I do. Do you do the same?
Now trying changing the default file server.
So I suppose there is a bug in ClearML.
yes everything that is downloaded is cached. The cache folder is in your config file :
` sdk {
# ClearML - default SDK configuration
storage {
cache {
# Defaults to system temp folder / cache
default_base_dir: "~/.clearml/cache"
size {
# max_used_bytes = -1
min_free_bytes = 10GB
# cleanup_margin_percent = 5%
}
}
direct_access: [
# Objects matching are considered to be available for direct access, i.e. they will not be downloaded
# or cached, and any download request will return a direct reference.
# Objects are specified in glob format, available for url and content_type.
{ url: "file://*" } # file-urls are always directly referenced
]
} `
Setting the api.files_server:
s3://myhost:9000/clearml in clearml.conf
works!