I also exposed elastic port and checked /_cluster/health/?level=shards
status is red
and this is the red shard:
"events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b": {
"status": "red",
"number_of_shards": 1,
"number_of_replicas": 0,
"active_primary_shards": 0,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 1,
"shards": {
"0": {
"status": "red",
"primary_active": false,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 1
}
}
},
@<1582904448076746752:profile|TightGorilla98> can you please share the entire log of the clearml-elastic container?
the container is running, how can I get more detailed status?
@<1523701087100473344:profile|SuccessfulKoala55> here is the log file, thanks!
found a corrupted file in the index, deleting this resolved the issue now everything is back to normal. thanks a lot for your help!
Hi @<1582904448076746752:profile|TightGorilla98> , can you check on the status of the elastic container?
additional context:
when i click on scalar charts i get the following message:
ERROR
Failed to get Scalar Charts
and in docker logs there appears an error:
...
clearml-elastic | "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
clearml-elastic | "at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
clearml-elastic | "at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
clearml-elastic | "at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
clearml-elastic | "at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
clearml-elastic | "at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]",
clearml-elastic | "at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]",
clearml-elastic | "at java.lang.Thread.run(Thread.java:1589) [?:?]",
clearml-elastic | "Caused by: org.elasticsearch.action.NoShardAvailableActionException",
clearml-elastic | "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:532) ~[elasticsearch-7.17.7.jar:7.17.7]",
clearml-elastic | "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:479) [elasticsearch-7.17.7.jar:7.17.7]",
clearml-elastic | "... 81 more"] }