@<1523701553372860416:profile|DrabOwl94> , I would suggest restarting the elastic container. If that doesn't help, check the ES folder permissions - maybe something changed

restarted ES container. same. no scalars

Just make sure you make regular backups

(wrong tab sorry :P)

can upgrading to 1.11.0 help?

{"type": "server", "timestamp": "2023-06-21T08:26:03,816Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "clearml", "message": "uncaught exception in thread [process reaper (pid 218)]", "cluster.uuid": "9l7JSn6ES0upH0UZK2C8Tw", "node.id": "x3JpeJiKSWSNsz8s7Y0Kuw" ,
uncaught exception in thread [process reaper (pid 218)]
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "modifyThread")
        at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)
        at java.base/java.security.AccessController.checkPermission(AccessController.java:1068)
        at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
        at org.elasticsearch.secure_sm.SecureSM.checkThreadAccess(SecureSM.java:160)
        at org.elasticsearch.secure_sm.SecureSM.checkAccess(SecureSM.java:120)
        at java.base/java.lang.Thread.checkAccess(Thread.java:2360)
        at java.base/java.lang.Thread.setDaemon(Thread.java:2308)
        at java.base/java.lang.ProcessHandleImpl.lambda$static$0(ProcessHandleImpl.java:103)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:637)
        at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:928)
        at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1021)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1589)
        at java.base/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)

yes

Having the latest versions is always a good practice

Did you check permissions?

only GPU and machine

Hi @<1523701553372860416:profile|DrabOwl94> , can you check if there are some errors in the Elastic container?