Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello. I'M Trying Out Self-Hosting With The Latest Version. Upon Spinning Up, Clearml-Elastic Errors-Out With:

Hello. I'm trying out self-hosting with the latest version. Upon spinning up, clearml-elastic errors-out with:

"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:173) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.18.jar:7.17.18]",
"at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.18.jar:7.17.18]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:1089) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.18.jar:7.17.18]",

The compose yml is unchanged, and checking with lsof , ports 9200 and 9300 are free. Changing the ports doesn't resolve the issue.
Any tips?

  
  
Posted one month ago
Votes Newest

Answers 11


based on my understanding, the key/secret is only for agent services on server side.

  
  
Posted one month ago

Should I create the credential at the localhost server ?

  
  
Posted one month ago

copy credential from webUI to ~/clearml.conf (linux)

  
  
Posted one month ago

first, you need to make sure all containers are running normally. especially the clearml-elastic , this container may restart every few seconds due to some errors. such situation may cause authorization issue.

  
  
Posted one month ago

Try below options:

sudo chown -R 1000:1000 /opt/clearml

in docker-compose.yaml , elasticsearch -> volumes
Change
- /opt/clearml/data/elastic_7:/usr/share/elasticsearch/data
TO
- /opt/clearml/data/elastic_7:/var/lib/elasticsearch/data

  
  
Posted one month ago

i have same issue. solved by option 2.

  
  
Posted one month ago

hope it’s helpful

  
  
Posted one month ago

also check apiserver and elastic logs and find if there are any errors.

  
  
Posted one month ago

Hey, thanks. I missed this in the docker configuration. This seems to resolve the problem, however now clearml-agent-services has authorization issues. In docker logs :

Unauthorized (invalid credentials) (failed to locate provided credentials)

I have set the generated credentials in my clearml conf file and the agent-services config in the compose.
image
image

  
  
Posted one month ago

i found the created credential will be missing.
image

  
  
Posted one month ago

then you delete the browser’s cookie and cache, log in webui again to check if the credential is still there.

  
  
Posted one month ago
126 Views
11 Answers
one month ago
one month ago
Tags