Hi VictoriousPenguin97 ,
This might indicate an issue with ElasticSearch (possible storage space watermark being reached) - can you check the ES container's log?

{"type": "server", "timestamp": "2021-12-22T10:56:48,500Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "0aOZYv7bQD--rcBaPQvSJQ", "node.id": "6hqtnbZLSyCot85eDVoHAw" } {"type": "server", "timestamp": "2021-12-22T10:56:48,500Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "0aOZYv7bQD--rcBaPQvSJQ", "node.id": "6hqtnbZLSyCot85eDVoHAw" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] }

all shards failed is never a good indication... is that the entire log?

There’s another log from elastic contatiner.
The logs is too long, I have to store them in txt file.

If you can locate the start of the log (or more errors in it), I might be able to understand what's going on - it seems some shards are not available, but I'm not sure why

I think these are all errors in container.
I also attached all logs in file.