Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Is There Any Documentation From Clearml On Best Practices For Mounting/Using External Ebs Volumes For The Clearml Server? We Would Like To Mount An External Ebs Volume To The

Is there any documentation from ClearML on best practices for mounting/using external EBS volumes for the ClearML server? We would like to mount an external EBS volume to the /opt/clearml/data directory and use it for storage, instead of the root EBS volume of the EC2 instance. However, we are running into some issues with this (of which there are greater details commented in this thread).

  
  
Posted 11 months ago
Votes Newest

Answers 8


Hi @<1594863216222015488:profile|ConvincingGrasshopper20> @<1541954607595393024:profile|BattyCrocodile47> , I am facing the same problem of Elasticsearch container, How did you resolve this. I am mounting EFS instead of EBS.

  
  
Posted 9 months ago

my approach was to spin up an EC2 and run the deployment there from within the EBS volume mount.

I symlinked /opt/clearml to /mnt/xvda/clearml to minimize docker-compose changes. been working out fine so far.

with aws-cdk, the deployment steps can be automated (format the volume, clone a repo with the config, etc). I can link you to a resource that may help with that if you're interested.

  
  
Posted 11 months ago

Like I said above, we want to mount an external EBS volume to the /opt/clearml/data directory. However, upon the creation of a net new EC2 instance using ClearML's community AMI, this directory already contains sub-directories and files due to the ClearML containers already being up and running. Thus, we can't mount an EBS volume to this directory until we (1) run a docker-compose down command to stop the running containers and (2) delete the pre-existing files found in the /opt/clearml/data directory. After having performed those two steps, we are able to mount an external EBS volume to the directory without issue and start the docker containers again with a docker-compose up command, which then repopulates the necessary sub-directories/files into the /opt/clearml/data .

The only problem we are facing with this approach is that the clearml-elastic container repeatedly fails to start up when following these steps, while the other containers have no issue. Why is that? I have attached container logs from the clearml-elastic container in the file below.

  
  
Posted 11 months ago

Ohh, btw how did you approach the problem of mounting ebs/efs where clearml-data and everything will be stored

  
  
Posted 9 months ago

@<1541954607595393024:profile|BattyCrocodile47> put together None

  
  
Posted 11 months ago

@<1593051292383580160:profile|SoreSparrow36> thank you for responding! Yes, could you share a link to that resource for us to look at? We are deploying our infrastructure using the AWS CDK so this is right in-line with what we'd be looking for.

  
  
Posted 11 months ago

Hey! Sorry, I don't think I ever solved this for elasticsearch 😕

  
  
Posted 9 months ago

I symlinked

/opt/clearml

to

/mnt/xvda/clearml

Genius! I don't think I accounted for making sure the volumes ended up in the EBS volume mount in this CDK example ^^^. And I modified the docker-compose.yml file to point at a different location. Sym-linking is totally the route I should take if I get time to come back and clean up this repo.

  
  
Posted 11 months ago
724 Views
8 Answers
11 months ago
9 months ago
Tags