Hi @<1523701601770934272:profile|GiganticMole91> , what is the exact version of Elasticsearch that is running now in your 1.15.0 installation? You can see it in the output of 'sudo docker ps'

This seems something different not connected to ES. Where do you get these logs?

In ES container please run "curl -XGET localhost:9200/_cat/indices"

I would try it again.

Hi @<1734020208089108480:profile|WickedHare16> , what is the image of the apiserver that you are running?

@<1734020208089108480:profile|WickedHare16> Do you mean that you see the plots now? Are there still any _attempt_serialize_numpy errors in the apiserver logs?

Please take a look here:
None
Does it match you scenario? Can you try the suggested workaround?

And are you still getting exactly this error?

<500/100: events.add_batch/v1.0 (General data error: err=1 document(s) failed to index., extra_info=[events-log-d1bd92a3b039400cbafc60a7a5b1e52b][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[events-log-d1bd92a3b039400cbafc60a7a5b1e52b][0]] containing [index {[events-log-d1bd92a3b039400cbafc60a7a5b1e52b][f3abecd0f46f4bd289e0ac39662fd850], source[{"timestamp":1747654820464,"type":"log","task":"fd3d00d99d88427bbc57...

No, it says that it does not detect any problematic shards. Given that output and the absence of the errors in the logs I would expect that you will not get the error anymore

What is the status that you get for the "events-log-d1bd92a3b039400cbafc60a7a5b1e52b" index?

Did you try to run your job again?

Probably the 9200 port is not mapped from the ES container in the docker compose
The easiest would be to perform "sudo docker exec -it clearml-elastic /bin/bash" and then run the curl command from inside the ES docker

Hi @<1835488771542355968:profile|PerplexedShells66> , please inspect your Elasticsearch logs. Any errors or warnings there?

Hi @<1523701601770934272:profile|GiganticMole91> , each scalar document in ES has a "task" field that is a task ID. The below query will show you the first 10 documents for the task ID:

curl -XGET "localhost:9200/<the scalar index name>/_search?q=task:<task ID>&pretty"

Hi @<1689446563463565312:profile|SmallTurkey79> , we identified the problem and working on the fix

Hi @<1578555761724755968:profile|GrievingKoala83> , the DELETED prefix in the model id means that the original model was already deleted. The reference that you see "__DELETED__63e920aeacb247c890c70e525576474c" does not point to any model but instead a reminder that there was a reference to 63e920aeacb247c890c70e525576474c model here but the model was removed

The path then would be as following:

• Upgrade the old deployment to the latest clearml server according to the clearml server upgrade procedure. This will automatically upgrade the data
• Backup your data folders (mongo and elastic)
• Deploy the latest clearml server on another machine and restore the data from the backup

Hi @<1526734383564722176:profile|BoredBat47> , it seems that your Elasticsearch version is out of sync with what the latest version of the apiserver requires (7.17.18). Can you please follow the instructions here to make sure that you use the latest images for the ClearML Server?
None

Can you please describe what working deployments you current have and what is you final goal?
Do you have an old deployment working or it was corrupted?
Do you want to upgrade that old deployment to a new one? Or you want to have a new deployment in some other place based on the data from the old deployment?

Can you please run 'sudo docker ps' and share the results? I want to see what is the real version of the apiserver image used

One of the most likely reasons for this issue would be insufficient free disk space for Elasticsearch. This may happen if less than 10% of free space is left on ES storage location. But there may be also other reasons

It depends on your usage. ES has some default watermarks that are activated when the amount of used space is above 85% and 90% (can be overwritten) of the storage. At some point it may transfer the index to a "readonly" state.

I do not see any issues in the log. Do you still get errors in the task due to the failure in events.add_batch?

Then possibly it is another reason. Need to search for in the ES logs

On what host did you run the curl command?

Yes

While on the host you can run some ES commands to check the shards health and allocations. For example this:

curl -XGET "localhost:9200/_cluster/allocation/explain?pretty"

It may give more clues to the problem

About the prefix part I think it should not matter. Just put your prefix instead of ' None .<ADDRESS>'

No. It is actually string concatenation. What you actually get is that an original string is broken into several parts. That are concatenated as following:
-d'{....' + ' + '....}'

I see now. It seems that the instructions that we provided updated only model urls and there are some more artifacts that need to be handled. Please try running the attached python script from inside your apiserver docker container. The script should fix all the task artifact links in mongo. Copy it to any place inside the running clearml-apiserver container and then run it as following:

python3 fix_mongo_urls.py --mongo-host 

 --host-source 

 --host-target http:...