Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey, Using K8S With Trains 0.16.1-320, All Of A Sudden The Entire Data (I.E Experiments, Tasks, Api Creds) Is Not Showing In The Ui Anymore. All Logs Seems To Be Fine Afai Can Tell... Any Idea What Went Wrong?

hey, using k8s with trains 0.16.1-320, all of a sudden the entire data (i.e experiments, tasks, API creds) is not showing in the UI anymore. All logs seems to be fine AFAI can tell... any idea what went wrong?

  
  
Posted 3 years ago
Votes Newest

Answers 30


No worries, and I hope you manage to get that backup.

  
  
Posted 3 years ago

(I mean new logs, while we are here did it report any progress)

  
  
Posted 3 years ago

ohh sec

  
  
Posted 3 years ago

Could it be it was never allocated to begin with ?

  
  
Posted 3 years ago

azure

  
  
Posted 3 years ago

but the PV seems to be just a path to the labeled node

  
  
Posted 3 years ago

check if you have any more of those recovery reports in the mongo log, it should report progress

I think I have sent you all the existing logs

  
  
Posted 3 years ago

And if this is the case, that would explain the empty elastic as well

  
  
Posted 3 years ago

Now I suspect what happened is it stayed on another node, and your k8s never took care of that

  
  
Posted 3 years ago

That somehow the PV never worked and it was all local inside the pod

  
  
Posted 3 years ago

Meaning the node restarted (or actually moved)

  
  
Posted 3 years ago

I wonder if it's completely lost

  
  
Posted 3 years ago

nothing for now

  
  
Posted 3 years ago

In that case, I think it is stuck on a previous Node, I can;t think of any other reason.
Do you have something else on the same PV that was lost ? like api server configuration?

  
  
Posted 3 years ago

so if the node went down and then some other node came up, the data is lost

  
  
Posted 3 years ago

yea the api server configuration also went away

okay that proves it

  
  
Posted 3 years ago

yea the api server configuration also went away

  
  
Posted 3 years ago

no that's for sure not

  
  
Posted 3 years ago

I will investigate a bit more and then check if I can recover

  
  
Posted 3 years ago

thank you for your time and support, I appreciate it!

  
  
Posted 3 years ago

🤞

  
  
Posted 3 years ago

I hope so

  
  
Posted 3 years ago

Oh dear, I think your theory might be correct, and this is just the mongo preallocating storage.
Which means the entire /opt/trains just disappeared

  
  
Posted 3 years ago

Now I suspect what happened is it stayed on another node, and your k8s never took care of that

that's an interesting theory

  
  
Posted 3 years ago

Could it be it was never allocated to begin with ?

what do you mean?

  
  
Posted 3 years ago

so if the node went down and then some other node came up, the data is lost

That might be the case. where is the k8s running ? cloud service ?

  
  
Posted 3 years ago

Damn 😞

  
  
Posted 3 years ago

backup?

  
  
Posted 3 years ago
1K Views
30 Answers
3 years ago
one year ago
Tags