Hi @<1523702496097210368:profile|ScantChimpanzee51> , I think this is more difficult. I think you would need to edit the urls in mongoDB per task/model/dataset
@<1523701070390366208:profile|CostlyOstrich36> thank you, now everything works so far!
Last thing: Is there any way to change all the links in the new ClearML server such that an artifact that was previous under s3://… is now taken from gs://… ? The actual data is already available under the gs:// link of course
Hi @<1523702496097210368:profile|ScantChimpanzee51> , your steps look ok but the error pretty much indicates that there is a folder permissions issue. Please navigate manually to /opt/clearml/data folder and check "ls -al" command what are the user and permissions for the "elastic_7" folder and then enter the elastic_7 folder and check the same for its "nodes" subfolder. If the permissions are correct try restarting the docker and checking if it helps.
More stack trace:
clearml-elastic | ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];
clearml-elastic | Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
clearml-elastic | at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
clearml-elastic | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
clearml-elastic | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
clearml-elastic | at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:398)
clearml-elastic | at java.base/java.nio.file.Files.createDirectory(Files.java:700)
clearml-elastic | at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
clearml-elastic | at java.base/java.nio.file.Files.createDirectories(Files.java:793)
clearml-elastic | at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:300)
clearml-elastic | at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:224)
clearml-elastic | at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:298)
clearml-elastic | at org.elasticsearch.node.Node.<init>(Node.java:427)
clearml-elastic | at org.elasticsearch.node.Node.<init>(Node.java:309)
clearml-elastic | at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
clearml-elastic | at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
clearml-elastic | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
clearml-elastic | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
clearml-elastic | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
clearml-elastic | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
clearml-elastic | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
clearml-elastic | at org.elasticsearch.cli.Command.main(Command.java:77)
clearml-elastic | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
clearml-elastic | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
clearml-elastic | For complete error details, refer to the log at /usr/share/elasticsearch/logs/clearml.log
clearml-elastic exited with code 1
clearml-apiserver | [2023-06-05 07:19:16,651] [10] [WARNING] [clearml.initialize] Could not connect to ElasticSearch Service. Retry 2 of 4. Waiting for 30sec
0 0 0 0 0 0 0 0 --:--:-- 0:00:34 --:--:-- 0curl: (7) Failed to connect to apiserver port 8008: No route to host
To recap, the server started up on GCP as expected before migrating the data over. The migration was done by
- deleting the current data
sudo rm -fR /opt/clearml/data/* - unpacking the backup
sudo tar -xzf ~/clearml_backup_data.tgz -C /opt/clearml/data - setting permissions
sudo chown -R 1000:1000 /opt/clearml