Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I'M Trying To Set Up Clearml Server On A New Vm But The Elasticsearch Container Is Erroring With The Following:

I'm trying to set up clearml server on a new vm but the elasticsearch container is erroring with the following:
Attaching to clearml-elastic clearml-elastic | Exception in thread "main" java.lang.RuntimeException: starting java failed with [1] clearml-elastic | output: clearml-elastic | [0.000s][error][logging] Error opening log file 'logs/gc.log': Permission denied clearml-elastic | [0.000s][error][logging] Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed. clearml-elastic | error: clearml-elastic | Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details. clearml-elastic | Error: Could not create the Java Virtual Machine. clearml-elastic | Error: A fatal exception has occurred. Program will exit. clearml-elastic | at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOption.java:119) clearml-elastic | at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(JvmOption.java:81) clearml-elastic | at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:38) clearml-elastic | at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:135) clearml-elastic | at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86) clearml-elastic | Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]I tried deleting everything and starting from scratch, but I always end up in this state. Can anyone help with this issue? Thanks

  
  
Posted one year ago
Votes Newest

Answers 11


WittyOwl57 what about? vm.max_map_count echo "vm.max_map_count=262144" > /tmp/99-clearml.conf
sudo mv /tmp/99-clearml.conf /etc/sysctl.d/99-clearml.conf
sudo sysctl -w vm.max_map_count=262144
sudo service docker restart `https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac (5)

  
  
Posted one year ago

Hi WittyOwl57
Are you starting a new server from scratch or is it running on previously stored data?

  
  
Posted one year ago

I actually put all the commands in a script. The failure mode is exactly the same. I have no idea what to do next.
` #!/bin/bash

clearml_root=$1

if [[ $# -gt 0 ]]; then
echo Using "$1" as root
else
echo No root argument was provided, using /datadrive1
clearml_root=/datadrive1
fi

clearml="$clearml_root/clearml"

rm -R "$clearml"

mkdir -p "$clearml"/data/elastic_7
mkdir -p "$clearml"/data/mongo_4/db
mkdir -p "$clearml"/data/mongo_4/configdb
mkdir -p "$clearml"/data/redis
mkdir -p "$clearml"/logs
mkdir -p "$clearml"/config
mkdir -p "$clearml"/data/fileserver

dc_yaml="$clearml"/docker-compose.yaml

curl -o "$dc_yaml"
sed -i 's#/opt/clearml#'"$clearml"'#g' "$dc_yaml"

chown -R 1000:1000 "$clearml"

docker-compose -f "$dc_yaml" down
docker-compose -f "$dc_yaml" up -d `

  
  
Posted one year ago

Hi SuccessfulKoala55 ,yes, I am running as sudo. It's not my first time setting this up either. Just that on this vm (which is a particularly large instance) I get this issue which has me stumped.

  
  
Posted one year ago

WittyOwl57 not much info here but since it seems to fail on the Java options, I suspect AgitatedDove14 is correct and the most likely cause is not enough RAM?

  
  
Posted one year ago

Hi AgitatedDove14 , I deleted everything in /opt/clearml as per the docs. Should I delete anything else?

  
  
Posted one year ago

Thanks AppetizingMouse58 . I managed to fix it by removing docker completely and reinstalling it.

  
  
Posted one year ago

WittyOwl57 could it be the EC2 instance is too small (i.e. not enough storage / memory) ?

  
  
Posted one year ago

I didn't add that to the script since the effect is persistent (i.e. it only needs to be done once, right?) In any case, I checked that multiple times and it was as expected.

  
  
Posted one year ago

WittyOwl57 maybe you have a filesystem permissions issue? Are you running this as root (or using sudo)?
Error opening log file 'logs/gc.log': Permission denied

  
  
Posted one year ago

Hi WittyOwl57 , there is a chance that the reason is in this setting: Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log ...
First it say says about invalid log option that may require further investigation. Second the file that it tries to write to is logs/gc.log and it is not under the $clearml folder where you give the write permissions to the ES user. I would try cancelling JVM logging at all or specifying the full path to the file so that it would be under the folder that has 1000:1000 owner. Please see more here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#gc-logging

  
  
Posted one year ago
1K Views
11 Answers
one year ago
one year ago
Tags