More stack trace:
clearml-elastic | ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];
clearml-elastic | Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
clearml-elastic | at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
clearml-elastic | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
clearml-elastic | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
clearml-elastic | at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:398)
clearml-elastic | at java.base/java.nio.file.Files.createDirectory(Files.java:700)
clearml-elastic | at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
clearml-elastic | at java.base/java.nio.file.Files.createDirectories(Files.java:793)
clearml-elastic | at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:300)
clearml-elastic | at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:224)
clearml-elastic | at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:298)
clearml-elastic | at org.elasticsearch.node.Node.<init>(Node.java:427)
clearml-elastic | at org.elasticsearch.node.Node.<init>(Node.java:309)
clearml-elastic | at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
clearml-elastic | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
clearml-elastic | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
clearml-elastic | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
clearml-elastic | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
clearml-elastic | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
clearml-elastic | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
clearml-elastic | at org.elasticsearch.cli.Command.main(Command.java:77)
clearml-elastic | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
clearml-elastic | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
clearml-elastic | For complete error details, refer to the log at /usr/share/elasticsearch/logs/clearml.log
clearml-elastic exited with code 1
clearml-apiserver | [2023-06-05 07:19:16,651] [10] [WARNING] [clearml.initialize] Could not connect to ElasticSearch Service. Retry 2 of 4. Waiting for 30sec
0 0 0 0 0 0 0 0 --:--:-- 0:00:34 --:--:-- 0curl: (7) Failed to connect to apiserver port 8008: No route to host
Hi @<1523702496097210368:profile|ScantChimpanzee51> , your steps look ok but the error pretty much indicates that there is a folder permissions issue. Please navigate manually to /opt/clearml/data folder and check "ls -al" command what are the user and permissions for the "elastic_7" folder and then enter the elastic_7 folder and check the same for its "nodes" subfolder. If the permissions are correct try restarting the docker and checking if it helps.
To recap, the server started up on GCP as expected before migrating the data over. The migration was done by
- deleting the current data
sudo rm -fR /opt/clearml/data/* - unpacking the backup
sudo tar -xzf ~/clearml_backup_data.tgz -C /opt/clearml/data - setting permissions
sudo chown -R 1000:1000 /opt/clearml
Hi @<1523702496097210368:profile|ScantChimpanzee51> , I think this is more difficult. I think you would need to edit the urls in mongoDB per task/model/dataset
@<1523701070390366208:profile|CostlyOstrich36> thank you, now everything works so far!
Last thing: Is there any way to change all the links in the new ClearML server such that an artifact that was previous under s3://… is now taken from gs://… ? The actual data is already available under the gs:// link of course