Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Have An Issue With Mongo Container When Try To Start Service. Here’S The Log From Mongo Container.

Hi,
I have an issue with mongo container when try to start service.
Here’s the log from mongo container.
clearml-mongo | 2021-12-21T08:24:54.567790371Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] db version v3.6.5 clearml-mongo | 2021-12-21T08:24:54.567795995Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] git version: a20ecd3e3a174162052ff99913bc2ca9a839d618 clearml-mongo | 2021-12-21T08:24:54.567800418Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016 clearml-mongo | 2021-12-21T08:24:54.567804483Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] allocator: tcmalloc clearml-mongo | 2021-12-21T08:24:54.567808327Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] modules: none clearml-mongo | 2021-12-21T08:24:54.567812042Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] build environment: clearml-mongo | 2021-12-21T08:24:54.567815800Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] distmod: debian81 clearml-mongo | 2021-12-21T08:24:54.567819707Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] distarch: x86_64 clearml-mongo | 2021-12-21T08:24:54.567823479Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] target_arch: x86_64 clearml-mongo | 2021-12-21T08:24:54.567827327Z 2021-12-21T08:24:54.567+0000 I CONTROL [initandlisten] options: { net: { bindIpAll: true }, setParameter: { internalQueryExecMaxBlockingSortBytes: "196100200" } } clearml-mongo | 2021-12-21T08:24:54.569488260Z 2021-12-21T08:24:54.569+0000 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty. clearml-mongo | 2021-12-21T08:24:54.570170433Z 2021-12-21T08:24:54.570+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. clearml-mongo | 2021-12-21T08:24:54.570464193Z 2021-12-21T08:24:54.570+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. clearml-mongo | 2021-12-21T08:24:54.570727002Z 2021-12-21T08:24:54.570+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7453M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), clearml-mongo | 2021-12-21T08:24:54.927700022Z 2021-12-21T08:24:54.927+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1640075094:927521][1:0x7ff713c77a00], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error clearml-mongo | 2021-12-21T08:24:54.927716016Z 2021-12-21T08:24:54.927+0000 E STORAGE [initandlisten] WiredTiger error (0) [1640075094:927545][1:0x7ff713c77a00], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata clearml-mongo | 2021-12-21T08:24:54.927718512Z 2021-12-21T08:24:54.927+0000 E STORAGE [initandlisten] WiredTiger error (0) [1640075094:927549][1:0x7ff713c77a00], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk clearml-mongo | 2021-12-21T08:24:54.927720532Z 2021-12-21T08:24:54.927+0000 E STORAGE [initandlisten] WiredTiger error (0) [1640075094:927569][1:0x7ff713c77a00], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options clearml-mongo | 2021-12-21T08:24:54.929247549Z 2021-12-21T08:24:54.929+0000 E - [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 416 clearml-mongo | 2021-12-21T08:24:54.929607136Z 2021-12-21T08:24:54.929+0000 I STORAGE [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating clearml-mongo | 2021-12-21T08:24:54.929613581Z 2021-12-21T08:24:54.929+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets... clearml-mongo | 2021-12-21T08:24:54.929615712Z 2021-12-21T08:24:54.929+0000 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock clearml-mongo | 2021-12-21T08:24:54.929619835Z 2021-12-21T08:24:54.929+0000 I CONTROL [initandlisten] now exiting clearml-mongo | 2021-12-21T08:24:54.929621612Z 2021-12-21T08:24:54.929+0000 I CONTROL [initandlisten] shutting down with code:100

  
  
Posted one year ago
Votes Newest

Answers 21


I saw somewhere that deleting the journaling files might help, but there's possible data loss

  
  
Posted one year ago

Yes, I backed up only Mongo path.

  
  
Posted one year ago

I’ll try that approach

  
  
Posted one year ago

Do you have backups?

  
  
Posted one year ago

Just for update. This issue fixed by
Start mongo container in bash mode. [sudo docker-compose -f docker-compose.yml run mongo bash] Run mongo db repair. [mongod --dbpath /data/db --repair] The DB is repaired successfully and restored.

  
  
Posted one year ago

That's great to hear VictoriousPenguin97 !

  
  
Posted one year ago

OS: Ubuntu
It works fine, but after sometimes MongoDB stops working. We find out that MongoDB is restarting from docker-compose ps.

  
  
Posted one year ago

And the error you posted above is the error you started getting in the middle of everything working?
Were there any errors prior?

  
  
Posted one year ago

Yes, It was working.
When this error occurred every experiment projects in Clearml UI are disappeared.

Then I did docker container ls ever containers work fine, except the mongo which pushed to “restarting” status.

The error attached in this post is logs from mongo container.

  
  
Posted one year ago

VictoriousPenguin97 this sounds like file-system corruption...

  
  
Posted one year ago

Aight. What OS are you on, also, what is the status of this deploy? Is this a clean install, version upgrade or it just stopped working after a restart? 🙂

  
  
Posted one year ago

Interesting, what version are you running?

  
  
Posted one year ago

mongo: 3.6.5, clearml-server: lastest

  
  
Posted one year ago

No, the run out of space occurred 3 day ago but MongoDB issue just occurred today.

  
  
Posted one year ago

From the error log it seems like an issue with the WiredTiger journal files, I wonder if that might be a related issue that only manifested a few days later

  
  
Posted one year ago

does a restart to the mongodb container help?

  
  
Posted one year ago

Restarting the MongoDB doesn’t help as well.

  
  
Posted one year ago

Actually run out of space was a issue, but we already move the clearml-storage to the bigger one.

  
  
Posted one year ago

Perhaps you've run out of space in the disk partition containing the data folders?

  
  
Posted one year ago

We also tried to change the engine from “WiredTiger” to “mmpav1",

  
  
Posted one year ago

did this issue occur roughly at the same time?

  
  
Posted one year ago
79 Views
21 Answers
one year ago
4 months ago
Tags