Yeah, they should:) The problem is that they are inside outer single quotes -d'{...}'

About the prefix part I think it should not matter. Just put your prefix instead of ' None .<ADDRESS>'

Hi @<1526734383564722176:profile|BoredBat47> , did the last update urls command work for you? I want to update our documentation

No. It is actually string concatenation. What you actually get is that an original string is broken into several parts. That are concatenated as following:
-d'{....' + ' + '....}'

Can you please share it as a text
It is indistinguishable visually but I want to make sure that each single quote is replaced with single quote slash single quote single quote
and not single quote slash double quote

Hi @<1523701601770934272:profile|GiganticMole91> , what is the exact version of Elasticsearch that is running now in your 1.15.0 installation? You can see it in the output of 'sudo docker ps'

Great! Thanks:)

The path then would be as following:

• Upgrade the old deployment to the latest clearml server according to the clearml server upgrade procedure. This will automatically upgrade the data
• Backup your data folders (mongo and elastic)
• Deploy the latest clearml server on another machine and restore the data from the backup

Hi @<1526734383564722176:profile|BoredBat47> , please try to remove the ", "case_insensitive": true" from the last line of the command that I sent to you

Can you please run 'sudo docker ps' and share the results? I want to see what is the real version of the apiserver image used

Please take a look here:
None
Does it match you scenario? Can you try the suggested workaround?

Thank you for the fix:) I will update the script for future usages

Hi @<1734020208089108480:profile|WickedHare16> , what is the image of the apiserver that you are running?

It depends on your usage. ES has some default watermarks that are activated when the amount of used space is above 85% and 90% (can be overwritten) of the storage. At some point it may transfer the index to a "readonly" state.

@<1734020208089108480:profile|WickedHare16> Do you mean that you see the plots now? Are there still any _attempt_serialize_numpy errors in the apiserver logs?

Please share you command

Hi @<1578555761724755968:profile|GrievingKoala83> , the DELETED prefix in the model id means that the original model was already deleted. The reference that you see "__DELETED__63e920aeacb247c890c70e525576474c" does not point to any model but instead a reminder that there was a reference to 63e920aeacb247c890c70e525576474c model here but the model was removed

Hi @<1526734383564722176:profile|BoredBat47> , it seems that your Elasticsearch version is out of sync with what the latest version of the apiserver requires (7.17.18). Can you please follow the instructions here to make sure that you use the latest images for the ClearML Server?
None

clearml:1 as well as clearml:latest point to the latest version which is 1.16.2:
None
This version is not able to work with the version of Elasticsearch that you use. I suggest to use the docker compose from the latest version that uses the updated versions of all the required infrastructure

@<1526734383564722176:profile|BoredBat47> So your upgraded your source deployment to the latest clearml server, then backed up the data and tried to restore it on the target deployment that runs the same (latest) clearml server, correct? What is the error that you see in elastic logs on the target deployment?

Hi @<1888397385122451456:profile|PreciousPigeon91> , what version of ClearML are you using?

In ES container please run "curl -XGET localhost:9200/_cat/indices"

What is the status that you get for the "events-log-d1bd92a3b039400cbafc60a7a5b1e52b" index?

Then possibly it is another reason. Need to search for in the ES logs

No, it says that it does not detect any problematic shards. Given that output and the absence of the errors in the logs I would expect that you will not get the error anymore

''' = ' + ' + '

Did you try to run your job again?

Hi @<1547028074090991616:profile|ShaggySwan64> , by default ES allocates up to 50% of total RAM. If you think that this is an overkill for your case then you can limit it with ES_JAVA_OPTS parameters. I think that the 2GB limit that we had their earlier is too restrictive and may affect the ES performance and stability. So try putting there the number that depends on your needs and available memory but I would not go with less than 4Gb

Hi @<1577468611524562944:profile|MagnificentBear85> , the instructions that you followed should fix the addresses of debug images. For artifacts there are different instructions. Please follow the instructions from "For artifacts" item:
[https://clear.ml/docs/latest/docs/faq/#debug-images-andor-artifacts-are-not-loading-in-the-u[…]clearml-server-to-a-new-address-how-do-i-fix-this----](https://clear.ml/docs/latest/docs/faq/#debug-images-andor-artifacts-are-not-loading-in-the-ui-after-i-migra...

Can you please share an example of the datasets wrong address? Where do you see it?