Like I said above, we want to mount an external EBS volume to the /opt/clearml/data directory. However, upon the creation of a net new EC2 instance using ClearML's community AMI, this directory already contains sub-directories and files due to the ClearML containers already being up and running. Thus, we can't mount an EBS volume to this directory until we (1) run a docker-compose down command to stop the running containers and (2) delete the pre-existing files found in the /opt/clearml/data directory. After having performed those two steps, we are able to mount an external EBS volume to the directory without issue and start the docker containers again with a docker-compose up command, which then repopulates the necessary sub-directories/files into the /opt/clearml/data .
The only problem we are facing with this approach is that the clearml-elastic container repeatedly fails to start up when following these steps, while the other containers have no issue. Why is that? I have attached container logs from the clearml-elastic container in the file below.
my approach was to spin up an EC2 and run the deployment there from within the EBS volume mount.
I symlinked /opt/clearml to /mnt/xvda/clearml to minimize docker-compose changes. been working out fine so far.
with aws-cdk, the deployment steps can be automated (format the volume, clone a repo with the config, etc). I can link you to a resource that may help with that if you're interested.
@<1593051292383580160:profile|SoreSparrow36> thank you for responding! Yes, could you share a link to that resource for us to look at? We are deploying our infrastructure using the AWS CDK so this is right in-line with what we'd be looking for.
@<1541954607595393024:profile|BattyCrocodile47> put together None
I symlinked
/opt/clearml
to
/mnt/xvda/clearml
Genius! I don't think I accounted for making sure the volumes ended up in the EBS volume mount in this CDK example ^^^. And I modified the docker-compose.yml file to point at a different location. Sym-linking is totally the route I should take if I get time to come back and clean up this repo.
Hi @<1594863216222015488:profile|ConvincingGrasshopper20> @<1541954607595393024:profile|BattyCrocodile47> , I am facing the same problem of Elasticsearch container, How did you resolve this. I am mounting EFS instead of EBS.
Hey! Sorry, I don't think I ever solved this for elasticsearch 😕
Ohh, btw how did you approach the problem of mounting ebs/efs where clearml-data and everything will be stored