Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Have An Issue With Mongo Container When Try To Start Service. Here’S The Log From Mongo Container.

Hi,
I have an issue with mongo container when try to start service.
Here’s the log from mongo container.
clearml-mongo | 2021-12-21T08:24:54.567790371Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] db version v3.6.5 clearml-mongo | 2021-12-21T08:24:54.567795995Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] git version: a20ecd3e3a174162052ff99913bc2ca9a839d618 clearml-mongo | 2021-12-21T08:24:54.567800418Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016 clearml-mongo | 2021-12-21T08:24:54.567804483Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] allocator: tcmalloc clearml-mongo | 2021-12-21T08:24:54.567808327Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] modules: none clearml-mongo | 2021-12-21T08:24:54.567812042Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] build environment: clearml-mongo | 2021-12-21T08:24:54.567815800Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] distmod: debian81 clearml-mongo | 2021-12-21T08:24:54.567819707Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] distarch: x86_64 clearml-mongo | 2021-12-21T08:24:54.567823479Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] target_arch: x86_64 clearml-mongo | 2021-12-21T08:24:54.567827327Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] options: { net: { bindIpAll: true }, setParameter: { internalQueryExecMaxBlockingSortBytes: "196100200" } } clearml-mongo | 2021-12-21T08:24:54.569488260Z 2021-12-21T08:24:54.569+0000 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty. clearml-mongo | 2021-12-21T08:24:54.570170433Z 2021-12-21T08:24:54.570+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. clearml-mongo | 2021-12-21T08:24:54.570464193Z 2021-12-21T08:24:54.570+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. clearml-mongo | 2021-12-21T08:24:54.570727002Z 2021-12-21T08:24:54.570+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7453M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), clearml-mongo | 2021-12-21T08:24:54.927700022Z 2021-12-21T08:24:54.927+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1640075094:927521][1:0x7ff713c77a00], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error clearml-mongo | 2021-12-21T08:24:54.927716016Z 2021-12-21T08:24:54.927+0000 E STORAGE [initandlisten] WiredTiger error (0) [1640075094:927545][1:0x7ff713c77a00], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata clearml-mongo | 2021-12-21T08:24:54.927718512Z 2021-12-21T08:24:54.927+0000 E STORAGE [initandlisten] WiredTiger error (0) [1640075094:927549][1:0x7ff713c77a00], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk clearml-mongo | 2021-12-21T08:24:54.927720532Z 2021-12-21T08:24:54.927+0000 E STORAGE [initandlisten] WiredTiger error (0) [1640075094:927569][1:0x7ff713c77a00], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options clearml-mongo | 2021-12-21T08:24:54.929247549Z 2021-12-21T08:24:54.929+0000 E - [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 416 clearml-mongo | 2021-12-21T08:24:54.929607136Z 2021-12-21T08:24:54.929+0000 I STORAGE [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating clearml-mongo | 2021-12-21T08:24:54.929613581Z 2021-12-21T08:24:54.929+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets... clearml-mongo | 2021-12-21T08:24:54.929615712Z 2021-12-21T08:24:54.929+0000 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock clearml-mongo | 2021-12-21T08:24:54.929619835Z 2021-12-21T08:24:54.929+0000 I CONTROL [initandlisten] now exiting clearml-mongo | 2021-12-21T08:24:54.929621612Z 2021-12-21T08:24:54.929+0000 I CONTROL [initandlisten] shutting down with code:100

  
  
Posted 2 years ago
Votes Newest

Answers 21


That's great to hear VictoriousPenguin97 !

  
  
Posted 2 years ago

Just for update. This issue fixed by
Start mongo container in bash mode. [sudo docker-compose -f docker-compose.yml run mongo bash] Run mongo db repair. [mongod --dbpath /data/db --repair] The DB is repaired successfully and restored.

  
  
Posted 2 years ago

Aight. What OS are you on, also, what is the status of this deploy? Is this a clean install, version upgrade or it just stopped working after a restart? 🙂

  
  
Posted 2 years ago

Actually run out of space was a issue, but we already move the clearml-storage to the bigger one.

  
  
Posted 2 years ago

We also tried to change the engine from “WiredTiger” to “mmpav1",

  
  
Posted 2 years ago

does a restart to the mongodb container help?

  
  
Posted 2 years ago

Yes, It was working.
When this error occurred every experiment projects in Clearml UI are disappeared.

Then I did docker container ls ever containers work fine, except the mongo which pushed to “restarting” status.

The error attached in this post is logs from mongo container.

  
  
Posted 2 years ago

OS: Ubuntu
It works fine, but after sometimes MongoDB stops working. We find out that MongoDB is restarting from docker-compose ps.

  
  
Posted 2 years ago

And the error you posted above is the error you started getting in the middle of everything working?
Were there any errors prior?

  
  
Posted 2 years ago

VictoriousPenguin97 this sounds like file-system corruption...

  
  
Posted 2 years ago

Restarting the MongoDB doesn’t help as well.

  
  
Posted 2 years ago

Yes, I backed up only Mongo path.

  
  
Posted 2 years ago

Interesting, what version are you running?

  
  
Posted 2 years ago

I’ll try that approach

  
  
Posted 2 years ago

No, the run out of space occurred 3 day ago but MongoDB issue just occurred today.

  
  
Posted 2 years ago

mongo: 3.6.5, clearml-server: lastest

  
  
Posted 2 years ago

Do you have backups?

  
  
Posted 2 years ago

did this issue occur roughly at the same time?

  
  
Posted 2 years ago

From the error log it seems like an issue with the WiredTiger journal files, I wonder if that might be a related issue that only manifested a few days later

  
  
Posted 2 years ago

I saw somewhere that deleting the journaling files might help, but there's possible data loss

  
  
Posted 2 years ago

Perhaps you've run out of space in the disk partition containing the data folders?

  
  
Posted 2 years ago
531 Views
21 Answers
2 years ago
one year ago
Tags