/opt/trains/data contains data stored by ElasticSearch and MongoDB, which you can't delete manually (this will break each service and Trains Server as well).
You can clean up files uploaded to Trains File Server which are located in /opt/trains/data/fileserver - deleting those will simply mean the WebUI will not be able to find them when trying to display experiment data, so if you have very old files related to old experiments that you don't need any more, you can remove them from that folder.
Experiments cleanup can be done using the APIClient tool (see here: https://github.com/allegroai/trains-server/issues/53#issuecomment-660667572 ) - this will clean up both MongoDB and ElasticSearch data.