About the prefix part I think it should not matter. Just put your prefix instead of ' None .<ADDRESS>'

@<1526734383564722176:profile|BoredBat47> Can you please share the exact command that you are running?

clearml:1 as well as clearml:latest point to the latest version which is 1.16.2:
None
This version is not able to work with the version of Elasticsearch that you use. I suggest to use the docker compose from the latest version that uses the updated versions of all the required infrastructure

Can you please run 'sudo docker ps' and share the results? I want to see what is the real version of the apiserver image used

Yes, it seems so. I though we were talking about the fresh installation that you did.

Hi @<1523701601770934272:profile|GiganticMole91> , I do not see any difference that could lead to Elasticsearch from v1.16 not being able to start from the data that was stored in v1.15. Probably some more information can be retrieved from ES logs right after the upgrade and services restart. If there are some reasons the prevent ES from loading the existing data they may be listed in the logs

Hi @<1577468611524562944:profile|MagnificentBear85> , the instructions that you followed should fix the addresses of debug images. For artifacts there are different instructions. Please follow the instructions from "For artifacts" item:
[https://clear.ml/docs/latest/docs/faq/#debug-images-andor-artifacts-are-not-loading-in-the-u[…]clearml-server-to-a-new-address-how-do-i-fix-this----](https://clear.ml/docs/latest/docs/faq/#debug-images-andor-artifacts-are-not-loading-in-the-ui-after-i-migra...

Hi @<1526734383564722176:profile|BoredBat47> , please try to remove the ", "case_insensitive": true" from the last line of the command that I sent to you

@<1526734383564722176:profile|BoredBat47> Please try running the following command

curl -XPOST -H 'Content-Type: application/json' 'localhost:9200/events-training_debug_image-*/_update_by_query?conflicts=proceed' -d'{
    "script": {
        "source": "ctx._source.url = ctx._source.url.replace('

.<OLD_ADDRESS>', '

.<NEW_ADDRESS>')",
        "lang": "painless"
    },
    "query": {"prefix": {"url": {"value": "

.<OLD_ADDRESS>", "case_insensitive": true...

''' = ' + ' + '

Hi @<1526734383564722176:profile|BoredBat47> , did the last update urls command work for you? I want to update our documentation

Yeah, they should:) The problem is that they are inside outer single quotes -d'{...}'

🙂

Thank you for the fix:) I will update the script for future usages

Thanks! We want to add the python script that I sent you to the next version of open source and change the instructions to use this script instead of copying the commands from web

Hi @<1523702496097210368:profile|ScantChimpanzee51> , according to the clearml logs the initial problem was due to the MongoDB not able to start with your data. The reason is that MongoDB cannot be jumped more than one version at a time. Your previous version used MongoDB 4.4 and the latest version uses MongoDB 6.0. In order to upgrade from you initial version to the latest one you should first upgrade to clearml version 1.17 that upgrades MongoDB from v4.4 to 5.0
Then you can upgrade to cle...

What is the status that you get for the "events-log-d1bd92a3b039400cbafc60a7a5b1e52b" index?

The path then would be as following:

• Upgrade the old deployment to the latest clearml server according to the clearml server upgrade procedure. This will automatically upgrade the data
• Backup your data folders (mongo and elastic)
• Deploy the latest clearml server on another machine and restore the data from the backup

Please let me know if it fixes the UI. I will ask to update our documentation then

Please take a look here:
None
Does it match you scenario? Can you try the suggested workaround?

@<1734020208089108480:profile|WickedHare16> Can you please share an example plot url that is not opened in UI but you can see it in a separate tab?

Hi @<1523701601770934272:profile|GiganticMole91> , what is the exact version of Elasticsearch that is running now in your 1.15.0 installation? You can see it in the output of 'sudo docker ps'

Can you please share an example of the datasets wrong address? Where do you see it?

Hi @<1578555761724755968:profile|GrievingKoala83> , the DELETED prefix in the model id means that the original model was already deleted. The reference that you see "__DELETED__63e920aeacb247c890c70e525576474c" does not point to any model but instead a reminder that there was a reference to 63e920aeacb247c890c70e525576474c model here but the model was removed

Hi @<1689446563463565312:profile|SmallTurkey79> , we identified the problem and working on the fix

Hi @<1734020208089108480:profile|WickedHare16> , what is the image of the apiserver that you are running?

Hi @<1547028074090991616:profile|ShaggySwan64> , by default ES allocates up to 50% of total RAM. If you think that this is an overkill for your case then you can limit it with ES_JAVA_OPTS parameters. I think that the 2GB limit that we had their earlier is too restrictive and may affect the ES performance and stability. So try putting there the number that depends on your needs and available memory but I would not go with less than 4Gb

I see now. It seems that the instructions that we provided updated only model urls and there are some more artifacts that need to be handled. Please try running the attached python script from inside your apiserver docker container. The script should fix all the task artifact links in mongo. Copy it to any place inside the running clearml-apiserver container and then run it as following:

python3 fix_mongo_urls.py --mongo-host 

 --host-source 

 --host-target http:...

This seems something different not connected to ES. Where do you get these logs?

Yes