Elaseticsearch spazzing out

I get the same when I copy /opt/clearml/data folder into /mnt/data/clearml/data

well, I connected to mongodb manually and it is empty, loaded with just examples

thanks for help

yes

I also dont have side panel for some reason

Getting errors in elastisearch when deleting tasks, get retunred "cant delete experiment"

Here are my clearml versions and elastisearch taking up 50GB

Im pretty sure

But it seems like the data is gone, not sure how to get them back

we use Ceph Storage Cluster, interface to it is the same as S3
I dont get what I have misconfigured.
The only thing I have not added is "region" field in clearml.conf because we literally dont have, its a self hosted cluster.
You can try and replicate this s3 config I have posted earlier.

It looks like im moving forward

Setting url in clearml.conf without "s3" as suggested works (But I dont add port ther, not sure if it breaks something, we dont have a port)
host: " our-host.com "

Then in test_task.py
task: clearml.Task = clearml.Task.init(
project_name="project",
task_name="task",
output_uri=" None ",
)

I think connection is created
What im getting now is bucket error, i suppose I have to specify it so...

@<1523701070390366208:profile|CostlyOstrich36> Any news on this? We are currently stuck without this fix, cant finish up clearml setup

I cant get the conf credentials to work
Specifying it like this gives me:
Exception has occurred: ValueError
Could not get access credentials for ' None ' , check configuration file ~/clearml.conf

from docker inspect I can see that allegorai/clearml uses:
"CLEARML_SERVER_VERSION=1.11.0",
"CLEARML_SERVER_BUILD=373"

Image hash:ed05631045c4237f59ad48f477e06dd72274ab67e70d2f9adc489431d1ce75d7

@<1523701435869433856:profile|SmugDolphin23> Any ideas how to fix this?

digging deeper it seems like a parsing issue

doesnt help

Sounds similar to our issue? We have self hosted S3
None

we are cleaning, but there is a major problem
When deleting a task from web UI, nothing is deleted elsewhere
Debug images are not deleted, models are not deleted. And I suspect that scalars and logs are not deleted too
Im not sure why is that so

maybe someone on your end can try to parse such a config and see if they also have the same problem

helper is returned as None for some reason

removing it doesnt fix the problem

s

Specifying it like this, gets me different error:

Exception has occurred: ValueError

• Insufficient permissions (delete failed) for None
  botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the DeleteObject operation: The me-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.

During handling of the above exception, another exception occurred:

File "/home/ma...

it looks like problem is the host field, whenever I add it I get:
2024-01-22 13:27:16,489 - clearml.storage - ERROR - Failed creating storage object None Reason: Missing key and secret for S3 storage access ( None )

Yes, credetials seems to work
Im trying to figure out not why I dont see the uploaded files / folders

• I checked maybe clearml task uses fileserver instead but i dont see any files in fileserver folder
• Nothing is uploaded in bucket (i will ask IT guy to check if im uploading any files in logs)