Mongodb – Restore mongoDB by –repair and WiredTiger

disaster recoverymongo-repairmongodbrecoverywiredtiger

We accidentally deleted the directory rm -rf /data/db
which was our MongoDB path, and thanks to
extundelete, we recovered it and got the directory /data/db.

Here are our files in the directory, and the files was generated under MongoDB version 3.4.

Folder diagnostic.data:

Folder journal:

Step 1: try to run mongod as normal

a) We ran mongod --port 27017 --dbpath /data/db --bind_ip_all and mongo, and expected there should have a user-defined database wecaXX, But, it did not show up.

> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB

In Robo3T

b) Then I tried to run mongod --port 27017 --dbpath /data/db --bind_ip_all --repair. The result was:

2019-03-25T14:10:02.170+0800 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] MongoDB starting : pid=23018 port=27017 dbpath=/data/db 64-bit host=iZj6c0pipuxk17pb7pbaw0Z
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] db version v4.0.7
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] git version: 1b82c812a9c0bbf6dc79d5400de9ea99e6ffa025
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] modules: none
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] build environment:
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten]     distmod: ubuntu1604
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten]     distarch: x86_64
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] options: { net: { bindIpAll: true, port: 27017 }, repair: true, storage: { dbPath: "/data/db" } }
2019-03-25T14:10:02.191+0800 W STORAGE  [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2019-03-25T14:10:02.194+0800 I STORAGE  [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-03-25T14:10:02.194+0800 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2019-03-25T14:10:02.194+0800 I STORAGE  [initandlisten]
2019-03-25T14:10:02.194+0800 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-03-25T14:10:02.194+0800 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-03-25T14:10:02.194+0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-03-25T14:10:02.818+0800 E STORAGE  [initandlisten] WiredTiger error (17) [1553494202:818725][23018:0x7f6119074a40], connection: __posix_open_file, 715: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1553494202:818725][23018:0x7f6119074a40], connection: __posix_open_file, 715: /data/db/WiredTiger.wt: handle-open: open: File exists
2019-03-25T14:10:02.818+0800 I STORAGE  [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.1
2019-03-25T14:10:03.832+0800 I STORAGE  [initandlisten] WiredTiger message [1553494203:832267][23018:0x7f6119074a40], txn-recover: Main recovery loop: starting at 4/11366912 to 5/256
2019-03-25T14:10:03.832+0800 I STORAGE  [initandlisten] WiredTiger message [1553494203:832674][23018:0x7f6119074a40], txn-recover: Recovering log 4 through 5
2019-03-25T14:10:03.898+0800 I STORAGE  [initandlisten] WiredTiger message [1553494203:898252][23018:0x7f6119074a40], txn-recover: Recovering log 5 through 5
2019-03-25T14:10:03.964+0800 I STORAGE  [initandlisten] WiredTiger message [1553494203:964880][23018:0x7f6119074a40], txn-recover: Set global recovery timestamp: 0
2019-03-25T14:10:03.998+0800 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2019-03-25T14:10:03.999+0800 E STORAGE  [initandlisten] WiredTiger error (17) [1553494203:999855][23018:0x7f6119074a40], WT_SESSION.create: __posix_open_file, 715: /data/db/_mdb_catalog.wt: handle-open: open: File exists Raw: [1553494203:999855][23018:0x7f6119074a40], WT_SESSION.create: __posix_open_file, 715: /data/db/_mdb_catalog.wt: handle-open: open: File exists
2019-03-25T14:10:04.000+0800 I STORAGE  [initandlisten] WiredTiger message unexpected file _mdb_catalog.wt found, renamed to _mdb_catalog.wt.1
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten]
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten]
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten]
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 3824 processes, 65535 files. Number of processes should be at least 32767.5 : 0.5 times number of files.
2019-03-25T14:10:04.020+0800 I STORAGE  [initandlisten] createCollection: admin.system.version with provided UUID: 47d8713d-ac61-4081-83bf-60209ad60a7d
2019-03-25T14:10:04.034+0800 W ASIO     [initandlisten] No TransportLayer configured during NetworkInterface startup
2019-03-25T14:10:04.036+0800 I COMMAND  [initandlisten] setting featureCompatibilityVersion to 4.0
2019-03-25T14:10:04.036+0800 I STORAGE  [initandlisten] repairDatabase admin
2019-03-25T14:10:04.037+0800 I STORAGE  [initandlisten] Repairing collection admin.system.version
2019-03-25T14:10:04.040+0800 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0-4352287918877967674. Not salvaging.
2019-03-25T14:10:04.048+0800 I INDEX    [initandlisten] build index on: admin.system.version properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.version" }
2019-03-25T14:10:04.048+0800 I INDEX    [initandlisten]          building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2019-03-25T14:10:04.055+0800 I STORAGE  [initandlisten] finished checking dbs
2019-03-25T14:10:04.055+0800 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2019-03-25T14:10:04.056+0800 I STORAGE  [initandlisten] Shutting down session sweeper thread
2019-03-25T14:10:04.057+0800 I STORAGE  [initandlisten] Finished shutting down session sweeper thread
2019-03-25T14:10:04.140+0800 I STORAGE  [initandlisten] shutdown: removing fs lock...
2019-03-25T14:10:04.140+0800 I CONTROL  [initandlisten] now exiting
2019-03-25T14:10:04.140+0800 I CONTROL  [initandlisten] shutting down with code:0

c) After repairing I re-ran mongod --port 27017 --dbpath /data/db --bind_ip_all, it still showed nothing (the same result as step a)).

Step 2: Use wiredTiger tool

Since that didn't work as I expected, I started to find any other tools or approach that may help me.
Here is the link Recovering a WiredTiger collection from a corrupt MongoDB installation which introduces how to use WiredTiger to recover collections.
And I decided to have a try.

I created a folder db_backup and put all my files into it. And I created a folder wiredtiger-3.0.0 under db_backup.I installed wiredtiger in folder wiredtiger-3.0.0:

a) To salvage the collection

root@iZj6c0pipuxk17pb7pbaw0Z:/data/db_backup/# cd ./wiredtiger-3.0.0
root@iZj6c0pipuxk17pb7pbaw0Z:/data/db_backup/wiredtiger-3.0.0# ./wt -v -h ../db_for_wt -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-23--360946650994865928.wt
        WT_SESSION.salvage 100

Based on the article above, it means that we have successfully salvaged this collection.

Error Occured:

b) To dump the collection

root@iZj6c0pipuxk17pb7pbaw0Z:/data/db_backup/wiredtiger-3.0.0# ./wt -v -h ../../db_backup -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R dump -f ../collection.dump collection-23--360946650994865928
lt-wt: cursor open(table:collection-23--360946650994865928) failed: No such file or directory

Note that the command above was indeed ended without .wt.

I have checked my directory argument and I found no fault. In the picture, the file collection-23--360946650994865928.wt is right Here.

The snapshot of collection-23--360946650994865928.wt just has been salvaged.

We could see some English or Chinese characters in it. And that data truly is from one of collection of database wecaXX.

Questions:

1) Now, I'm stuck by dumping the collection. Does anyone know what's wrong with that?

2) collection-23--360946650994865928.wt contains our most important data. If we could not restore the whole database, extracting data from that file will be still very useful. However, if we do copy paste manually, not all the characters are well written; they still need to be decrypted. Does anyone know how to do that?

3) is it possible that we did not recover all the files (or the entire body of all the files) under /data/db?

Any comment or answer will be appreciated!!

Best Answer

As per your screenshot of Robo3T where only showing the System database (i.e., admin & local). There is no any user defined database is available with your connected host.

Eventhough the files you have generated under the mongodb version is 3.4 & the starting of MongoDB db version is sppearing as v4.0.7.

I will somewhere guess that you have been loose your user created database.

Suppose that if user database wil be there then automatically it will after host connection with Robo 3T-1.2 tool.

For example as the screenshot of Robo 3T - 1.1 appears in my environment.

For verification you can also check with mongo shell, after connecting the mongo shell you can check the command such as show dbs;

It will apears like that

> show dbs
DynamicsAX  0.006GB
ERP         0.086GB
admin       0.000GB
blog        0.000GB
config      0.000GB
enron       0.207GB
local       0.000GB
mflix       0.060GB
test        0.001GB

Note:- Right Now, Before doing any rocket science first you have to take mongodump of your data directory .

As per mention screenshot by you here the database Recovery is a user defined database.

As per your error mongodump error log

root@iZj6c0pipuxk17pb7pbaw0Z:/data/db_backup/wiredtiger-3.0.0# ./wt -v -h ../../db_backup -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R dump -f ../collection.dump collection-23--360946650994865928
lt-wt: cursor open(table:collection-23--360946650994865928) failed: No such file or directory

It's unable to mongodump successfully.

Note: The mongodump will fail if you’re using a version of MongoDB < 3.2, as 3.2 is built against WiredTiger 2.7.

After edit the question of OP

Now, I'm stuck during dumping the collection. Are there anything wrong with me?

Yes, Because the mongodump not went successfully.

In the collection-23--360946650994865928.wt, have we really got the whole data? Even if we could not recover the database, Could we extract data from these *.wt files?

As here already community wiki has been updated the answer, the *.wt are binary data files used by the WiredTiger storage engine. Individual files are not usable as a standalone backup. If you want to take a file copy backup of a MongoDB dbPath you need to include all of the files using a Supported File Copy Backup Method.

for further you ref here , here , here , here and here

Exporting the users

Assuming you have already upgraded all of your config servers to WiredTiger, here are some steps to add the user information:

Disable the balancer
Stop the last config server listed in your mongos' configDB setting (will call that config3 for the purpose of these steps). This will ensure your sharded cluster metadata remains read-only for the following steps.
Re-start config2 using the mmap data directory

At this stage you should have:
- config1 (running WiredTiger)
- config2 (running mmap with user/role data)
- config3 (stopped)
Export the data from config2:

mongodump --db config --dumpDbUsersAndRoles --username .. --password ..

Add any other parameters needed, eg --authenticationDatabase .. if you need to auth against another database.
If you have users in the admin database on your config server, you will also want to dump that as well.
(optional) Remove files from your dump except for the user/role information. If you are certain nothing has changed since you did the original migration from mmap to WiredTiger you could skip this step, however it would be safer to not overwrite any existing data.

Preview the files to remove:

find ./dump -type f -not -name "\$admin.system*"

WARNING: removing files, make sure you have previewed to confirm:

find ./dump -type f -not -name "\$admin.system*" | xargs rm
Re-start config2 using the wiredTiger storage engine
Run: mongorestore --db config --restoreDbUsersAndRoles dump/config/

You should see messages about restoring users & roles, for example:

2015-03-18T02:41:34.887+1100 restoring users from dump/config/$admin.system.users.bson

2015-03-18T02:41:34.887+1100 restoring roles from dump/config/$admin.system.roles.bson
Login to config2 and confirm the users are correctly setup (i.e. auth with admin account, use db.getUsers() to check).

At this stage you should have:
- config1 (running WiredTiger)
- config2 (running WiredTiger with user/role data)
- config3 (stopped)
Copy the dump directory to config1 and repeat the mongorestore step.
Shutdown config2 (to keep the sharded cluster metadata readonly for the next step).

At this stage you should have:
- config1 (running WiredTiger with user/role data)
- config2 (stopped)
- config3 (stopped)
Start config3. Copy the dump directory to config3, and repeat the mongorestore step.

At this stage you should have:
- config1 (running WiredTiger with user/role data)
- config2 (stopped)
- config3 (running WiredTiger with user/role data)
Start config2. At this point all config servers should be online with the user information.
Re-enable the balancer so normal balancing activity & chunk migration can resume.

Mongodb – ‘mongod wiredTiger’ on Ubuntu

I wasn't able to use wiredTiger upgrading MongoDB. However, at May 17 I uninstalled MongoDB 2.6 then installed MongoDB 3.0.3 . Immediate after the installation, I added storageEngine=wiredTiger on top of my mongod.conf file. Then I gave sudo service mongod start command and eventually I could.

Edit:

For fresh installed as directed by official documentation;

Open configuration file using sudo nano /etc/mongod.conf
Change the # engine line to engine: wiredTiger like the below
Run mongod using the command sudo service mongod start

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
  engine: wiredTiger
#  mmapv1:
#  wiredTiger:

Edit:

If the current version is supported the wiredTiger;

Get the backup of the current database using mongodump command
Stop the mongod service using sudo service mongod stop command
Add storageEngine=wiredTiger text as the first line of mongod.conf file
Delete the all file on /var/lib/mongodb (or /data/db folder if used)
[This is important. Because MongoDB cannot convert the current MMAP db files to wiredTiger format]
Start the mongod service using sudo service mongod start command
Restore the database from the backup using [mongorestore][2] command
wiredTiger is being used...