Hello. I'M Trying Out Self-Hosting With The Latest Version. Upon Spinning Up, Clearml-Elastic Errors-Out With:

Answered

Hello. I'm trying out self-hosting with the latest version. Upon spinning up, clearml-elastic errors-out with:

"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:173) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.18.jar:7.17.18]",
"at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.18.jar:7.17.18]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:1089) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.18.jar:7.17.18]",

The compose yml is unchanged, and checking with lsof , ports 9200 and 9300 are free. Changing the ports doesn't resolve the issue.
Any tips?

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					VastGiraffe70
				
					0
					 × 1

Votes Newest

Answers 11

first, you need to make sure all containers are running normally. especially the clearml-elastic , this container may restart every few seconds due to some errors. such situation may cause authorization issue.

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					StrongSeaturtle89
				
					0
					 × 1

hope it’s helpful

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					StrongSeaturtle89
				
					0
					 × 1

Try below options:

sudo chown -R 1000:1000 /opt/clearml

in docker-compose.yaml , elasticsearch -> volumes
Change
- /opt/clearml/data/elastic_7:/usr/share/elasticsearch/data
TO
- /opt/clearml/data/elastic_7:/var/lib/elasticsearch/data

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					StrongSeaturtle89
				
					0
					 × 1

based on my understanding, the key/secret is only for agent services on server side.

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					StrongSeaturtle89
				
					0
					 × 1

i found the created credential will be missing.

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					StrongSeaturtle89
				
					0
					 × 1

Hey, thanks. I missed this in the docker configuration. This seems to resolve the problem, however now clearml-agent-services has authorization issues. In docker logs :

Unauthorized (invalid credentials) (failed to locate provided credentials)

I have set the generated credentials in my clearml conf file and the agent-services config in the compose.

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					VastGiraffe70
				
					0
					 × 1

copy credential from webUI to ~/clearml.conf (linux)

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					StrongSeaturtle89
				
					0
					 × 1

Should I create the credential at the localhost server ?

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					VastGiraffe70
				
					0
					 × 1

also check apiserver and elastic logs and find if there are any errors.

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					StrongSeaturtle89
				
					0
					 × 1

then you delete the browser’s cookie and cache, log in webui again to check if the credential is still there.

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					StrongSeaturtle89
				
					0
					 × 1

i have same issue. solved by option 2.

  				
Posted 
	7 months ago

					More
				  		
  Report
		
					StrongSeaturtle89
				
					0
					 × 1

Write your answer

472 Views

11 Answers

7 months ago