clearml.conf is a fresh one i did clearml-init to make sure

I tried it with port, but still having the same issue
Tried it with/without secure and multipart

digging deeper it seems like a parsing issue

The problem is that clearml.conf s3 config doesnt support empty region field, even empty strings crashes it

File is written

we use Ceph Storage Cluster, interface to it is the same as S3
I dont get what I have misconfigured.
The only thing I have not added is "region" field in clearml.conf because we literally dont have, its a self hosted cluster.
You can try and replicate this s3 config I have posted earlier.

maybe someone on your end can try to parse such a config and see if they also have the same problem

will it be appended in clearml?
"s3" is part of domain to the host

removing it doesnt fix the problem

1. is 50GB elastisearch normal? Have you seen it. elsewhere or are we doing something wrong, one thing I think is that we are probably logging too frequently
2. Is it possible to somehow clean up this?

I also have noticed that this incident usually happens in the morning at around 6-7AM
Are there maybe some clearnup tasks or backups running on clearml server at those times?

What do you mean by reusing the task for clearml Dataset, got a code example?
We have multiple different projects with multiple people working on each project.
This is our most used code on dataset uploading

It is also possible to just make a copy of all the database files and move them to another server

@<1523701070390366208:profile|CostlyOstrich36> Hello John, we are still unable to use clearml with our self hosted s3 CEPH instances, is there any update on the hotfix for 1.14?

ClearML team should really write up some tutorial about this. I see this question weekly now. The short answer on what we did when we migrated servers was to wite a python script that takes data from clearml mongodb(stores tasks and datasets) and elastic (stores debug image urls, logs, scalars) and migrate them to other clearml instance databases