Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey, I Am Trying To Move The Fileserver To S3. As Here:

Hey,
I am trying to move the fileserver to S3.
As here: https://github.com/allegroai/trains-server/issues/35
I update the trains.conf with my s3 bucket and the s3 credentials.
and it works ๐Ÿ™‚ the debug)images are uploaded there.

But, when I am trying to see the images in Trains-server UI I am getting: "Please provide credentials for bucketย XXXX"
Can I give those credentials for all the users or for the server? so we won`t need to enter the credentials for each user?

Thanks

  
  
Posted 4 years ago
Votes Newest

Answers 20


As long as you still have the file server running as part of Trains, and the file server can reach the files using /opt/trains/data/fileserver - it should be OK

  
  
Posted 4 years ago

No, because the UI will still try to access http://<server-ip>:8081/... for old debug images - it won't try to access the S3

  
  
Posted 4 years ago

Regarding of moving the fileserver to S3, what is the best way to move the old data to S3 ?

I think if I will move all the /opt/trains/data/fileserver to s3,
the trains-server will not know that right?

  
  
Posted 4 years ago

SuccessfulKoala55 Thanks ๐Ÿ™ I will give it a try tomorrow ๐Ÿ™‚

  
  
Posted 4 years ago

CooperativeFox72 Logs and graphs are not stored on the fileserver - only models/artifacts (if uploaded there) and debug images (by default). For both, you have files on the fileserver (which can be moved on the file system) but you also have the URL itself registered in Elastic and Mongo, so you can't just move the file, since the UI and Trains won't find them if they try to access them using the URL stored in the metadata - you'll have to edit the metadata for that...

  
  
Posted 4 years ago

So for the UI each user will need to add the credentials only once, the users should have the credentials in ~trains.conf file. They just need to copy those to the profile page

  
  
Posted 4 years ago

like that I won't need to change metadata

  
  
Posted 4 years ago

Hi CooperativeFox72 , I just realized I might have misunderstood you - if you mount /opt/trains/data/fileserver to some S3 bucket and keep using the fileserver configuration in your trains.conf , it should absolutely work....

  
  
Posted 4 years ago

Thanks for the reply,
I saw that it prefer to change the fileserver in trains.conf to s3://XXX
So, I changed this as I wrote before.

  
  
Posted 4 years ago

Thanks I just want to avoid giving the credentials to every user.
If it won't possible, I will do it..

  
  
Posted 4 years ago

Hi CooperativeFox72 , moving old data to a different storage would require internal editing of the URL metadata registered in the events already stored in the system... It's possible, but not trivial. If you'd like, I can explain the basic principle... We're talking about debug images, right?

  
  
Posted 4 years ago

Hi SuccessfulKoala55 , yes for now I will like to start moving what inside the /opt/trains/data/fileserver..
because as I understand the logs and graphs are saved in elastic so I think it will not be easy to move them as well right?

  
  
Posted 4 years ago

I was under the impression you wanted to stop using the Trains fileserver, but now I realize you might just want to move the storage away from your local file system ๐Ÿ™‚

  
  
Posted 4 years ago

If I will mount the S3 bucket to the trains-server and link the mount to /opt/trains/data/fileserver does it will work?

  
  
Posted 4 years ago

Ohh I understood, so can you give me a short explanation on how to change the meta data?

  
  
Posted 4 years ago

SuccessfulKoala55

  
  
Posted 4 years ago

Hi CooperativeFox72 ,

basically each user should do it once, just to add credentials in the userโ€™s profile page.

About credentials for the server, let me check that for you

  
  
Posted 4 years ago

Basically you'll need to send a PORT request to ElasticSearch - I'll send an example here soon ๐Ÿ™‚

  
  
Posted 4 years ago

Does it still work if I will keep trains.conf like this, and mount the S3 also?

  
  
Posted 4 years ago

Ok thanks for the really fast answer ๐Ÿ™‚

  
  
Posted 4 years ago
1K Views
20 Answers
4 years ago
one year ago
Tags
Similar posts