While on the host you can run some ES commands to check the shards health and allocations. For example this:

curl -XGET "localhost:9200/_cluster/allocation/explain?pretty"

It may give more clues to the problem

On what host did you run the curl command?

And are you still getting exactly this error?

<500/100: events.add_batch/v1.0 (General data error: err=1 document(s) failed to index., extra_info=[events-log-d1bd92a3b039400cbafc60a7a5b1e52b][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[events-log-d1bd92a3b039400cbafc60a7a5b1e52b][0]] containing [index {[events-log-d1bd92a3b039400cbafc60a7a5b1e52b][f3abecd0f46f4bd289e0ac39662fd850], source[{"timestamp":1747654820464,"type":"log","task":"fd3d00d99d88427bbc57...

What is the status that you get for the "events-log-d1bd92a3b039400cbafc60a7a5b1e52b" index?

Thanks! We want to add the python script that I sent you to the next version of open source and change the instructions to use this script instead of copying the commands from web

Hi @<1523701601770934272:profile|GiganticMole91> , what is the exact version of Elasticsearch that is running now in your 1.15.0 installation? You can see it in the output of 'sudo docker ps'

@<1734020208089108480:profile|WickedHare16> Do you mean that you see the plots now? Are there still any _attempt_serialize_numpy errors in the apiserver logs?

I would try it again.

Then possibly it is another reason. Need to search for in the ES logs

Hi @<1835488771542355968:profile|PerplexedShells66> , please inspect your Elasticsearch logs. Any errors or warnings there?

Yes, it seems so. I though we were talking about the fresh installation that you did.

Hi @<1547028074090991616:profile|ShaggySwan64> , by default ES allocates up to 50% of total RAM. If you think that this is an overkill for your case then you can limit it with ES_JAVA_OPTS parameters. I think that the 2GB limit that we had their earlier is too restrictive and may affect the ES performance and stability. So try putting there the number that depends on your needs and available memory but I would not go with less than 4Gb

Can you please share an example of the datasets wrong address? Where do you see it?

It depends on your usage. ES has some default watermarks that are activated when the amount of used space is above 85% and 90% (can be overwritten) of the storage. At some point it may transfer the index to a "readonly" state.

Hi @<1526734383564722176:profile|BoredBat47> , please try to remove the ", "case_insensitive": true" from the last line of the command that I sent to you

Hi @<1523701601770934272:profile|GiganticMole91> , I do not see any difference that could lead to Elasticsearch from v1.16 not being able to start from the data that was stored in v1.15. Probably some more information can be retrieved from ES logs right after the upgrade and services restart. If there are some reasons the prevent ES from loading the existing data they may be listed in the logs

This seems something different not connected to ES. Where do you get these logs?

Can you please describe what working deployments you current have and what is you final goal?
Do you have an old deployment working or it was corrupted?
Do you want to upgrade that old deployment to a new one? Or you want to have a new deployment in some other place based on the data from the old deployment?

Probably the 9200 port is not mapped from the ES container in the docker compose
The easiest would be to perform "sudo docker exec -it clearml-elastic /bin/bash" and then run the curl command from inside the ES docker

Thank you for the fix:) I will update the script for future usages

Yeah, they should:) The problem is that they are inside outer single quotes -d'{...}'

@<1526734383564722176:profile|BoredBat47> Please try running the following command

curl -XPOST -H 'Content-Type: application/json' 'localhost:9200/events-training_debug_image-*/_update_by_query?conflicts=proceed' -d'{
    "script": {
        "source": "ctx._source.url = ctx._source.url.replace('

.<OLD_ADDRESS>', '

.<NEW_ADDRESS>')",
        "lang": "painless"
    },
    "query": {"prefix": {"url": {"value": "

.<OLD_ADDRESS>", "case_insensitive": true...

@<1523701601770934272:profile|GiganticMole91> Is your ES deployment a single node or a cluster? If you compare the elaststicsearch section of the docker compose of your currently working version (1.15.0) and then one that you tried to install (v1.16) do you see any difference?

Please take a look here:
None
Does it match you scenario? Can you try the suggested workaround?

Great! Thanks:)

Hi @<1734020208089108480:profile|WickedHare16> , what is the image of the apiserver that you are running?

@<1734020208089108480:profile|WickedHare16> Can you please share an example plot url that is not opened in UI but you can see it in a separate tab?

Yes

One of the most likely reasons for this issue would be insufficient free disk space for Elasticsearch. This may happen if less than 10% of free space is left on ES storage location. But there may be also other reasons

About the prefix part I think it should not matter. Just put your prefix instead of ' None .<ADDRESS>'