Mongodb – How to repair mongodb service after an upgrade to Fedora 28

mongodbupgrade

I just upgraded to Fedora 28 and now my mongod service doesn't launch anymore.

One possible solution that would fork in my case would be to be able to downgrade mongodb to 3.4, as the system upgrade moved it to 3.6.

Bellow is a report of an shell session aiming at gathering data on this issue:

▶ sudo systemctl start mongod                     
Job for mongod.service failed because the control process exited with error code.
# systemctl status mongod.service
See "systemctl status mongod.service" and "journalctl -xe" for details.
● mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2018-05-03 16:47:11 CEST; 9min ago
  Process: 5132 ExecStart=/bin/sh -c exec $NUMACTL /usr/bin/mongod $OPTIONS run (code=exited, status=100)

May 03 16:47:10 localhost.localdomain systemd[1]: Starting High-performance, schema-free document-oriented database...
May 03 16:47:10 localhost.localdomain sh[5132]: about to fork child process, waiting until server is ready for connections.
May 03 16:47:10 localhost.localdomain sh[5132]: forked process: 5134
May 03 16:47:11 localhost.localdomain sh[5132]: ERROR: child process failed, exited with error number 100
May 03 16:47:11 localhost.localdomain sh[5132]: To see additional information in this output, start without the "--fork" option.
May 03 16:47:11 localhost.localdomain systemd[1]: ^[[0;1;39m^[[0;1;39mmongod.service: Control process exited, code=exited status=100
May 03 16:47:11 localhost.localdomain systemd[1]: ^[[0;1;39m^[[0;1;39mmongod.service: Failed with result 'exit-code'.
May 03 16:47:11 localhost.localdomain systemd[1]: ^[[0;1;39mFailed to start High-performance, schema-free document-oriented database.

▶ grep path /etc/mongod.conf
  path: /var/log/mongodb/mongod.log
  # Append to logpath instead of over-writing (false by default)
  # Full path to pidfile (if not set, no pidfile is created)
    pathPrefix: /var/run/mongodb


▶ sudo chown mongodb /var/lib/mongodb

▶ sudo mongod --fork --logpath /var/log/mongodb.log
[sudo] Mot de passe de psychoslave : 
about to fork child process, waiting until server is ready for connections.
forked process: 11983
ERROR: child process failed, exited with error number 100
To see additional information in this output, start without the "--fork" option.

▶ sudo mongod --dbpath=/var/lib/mongodb                                                                                       [130/222]
2018-05-03T16:32:19.824+0200 I CONTROL  [initandlisten] MongoDB starting : pid=29148 port=27017 dbpath=/var/lib/mongodb 64-bit host=loc
alhost.localdomain                                                                                                               
2018-05-03T16:32:19.824+0200 I CONTROL  [initandlisten] db version v3.6.3                                                             
2018-05-03T16:32:19.824+0200 I CONTROL  [initandlisten] git version: nogitversion                                                 
2018-05-03T16:32:19.824+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.0h-fips  27 Mar 2018                   
2018-05-03T16:32:19.824+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2018-05-03T16:32:19.825+0200 I CONTROL  [initandlisten] modules: none                                                                  
2018-05-03T16:32:19.825+0200 I CONTROL  [initandlisten] build environment:                                                             
2018-05-03T16:32:19.825+0200 I CONTROL  [initandlisten]     distarch: x86_64                         
2018-05-03T16:32:19.825+0200 I CONTROL  [initandlisten]     target_arch: x86_64                                                 
2018-05-03T16:32:19.825+0200 I CONTROL  [initandlisten] options: { storage: { dbPath: "/var/lib/mongodb" } }
2018-05-03T16:32:19.825+0200 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage eng
ine, so setting the active storage engine to 'wiredTiger'.                                                                            
2018-05-03T16:32:19.825+0200 I STORAGE  [initandlisten]                                                                                
2018-05-03T16:32:19.825+0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTige
r storage engine                                                                                                                       
2018-05-03T16:32:19.825+0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem           
2018-05-03T16:32:19.825+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=5444M,session_max=20000,eviction=(thr
eads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manag
er=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),                                                      
2018-05-03T16:32:20.628+0200 I STORAGE  [initandlisten] WiredTiger message [1525357940:628222][29148:0x7fb1dd26fa80], txn-recover: Main
 recovery loop: starting at 648/768                                                                                                   
2018-05-03T16:32:20.698+0200 I STORAGE  [initandlisten] WiredTiger message [1525357940:698440][29148:0x7fb1dd26fa80], txn-recover: Reco
vering log 648 through 649                                                                                                             
2018-05-03T16:32:20.785+0200 I STORAGE  [initandlisten] WiredTiger message [1525357940:785494][29148:0x7fb1dd26fa80], txn-recover: Reco
vering log 649 through 649                                                                                                             
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten]                                                                                 
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.                   
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recomme
nded.                                                                                                                                  
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten]           
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten] ** WARNING: This server is bound to localhost.
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten] **          Remote systems will be unable to connect to this server.
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten] **          Start the server with --bind_ip <address> to specify which IP
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten] **          addresses it should serve responses from, or with --bind_ip_all to
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten] **          bind to all interfaces. If this behavior is desired, start the
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten] **          server with --bind_ip 127.0.0.1 to disable this warning.
2018-05-03T16:32:21.569+0200 I CONTROL  [initandlisten]
2018-05-03T16:32:21.686+0200 F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to vers
ion 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.
2018-05-03T16:32:21.686+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-05-03T16:32:21.686+0200 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-05-03T16:32:21.686+0200 I REPL     [initandlisten] shutdown: removing all drop-pending collections...
2018-05-03T16:32:21.686+0200 I REPL     [initandlisten] shutdown: removing checkpointTimestamp collection...
2018-05-03T16:32:21.687+0200 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2018-05-03T16:32:22.610+0200 I STORAGE  [initandlisten] WiredTiger message [1525357942:610149][29148:0x7fb1dd26fa80], txn-recover: Main
 recovery loop: starting at 649/4096
2018-05-03T16:32:22.681+0200 I STORAGE  [initandlisten] WiredTiger message [1525357942:681792][29148:0x7fb1dd26fa80], txn-recover: Reco
vering log 649 through 650
2018-05-03T16:32:22.762+0200 I STORAGE  [initandlisten] WiredTiger message [1525357942:762498][29148:0x7fb1dd26fa80], txn-recover: Reco
vering log 650 through 650
2018-05-03T16:32:23.549+0200 I STORAGE  [initandlisten] shutdown: removing fs lock...
2018-05-03T16:32:23.550+0200 I CONTROL  [initandlisten] now exiting
2018-05-03T16:32:23.550+0200 I CONTROL  [initandlisten] shutting down with code:62

And that's basically where I'm stuck right now, as the documentation on upgrade process assume that one can actually run the server and run a command like

db.adminCommand( { setFeatureCompatibilityVersion: "3.6" } )

The obvious idea for downgrading would be to use the package manager to do so, but this won't work:

▶ sudo dnf downgrade mongodb
Package mongodb of lowest version already installed, cannot downgrade it.
Error: No packages marked for downgrade.

On the other hand, dnf reinstall mongodb seems not enough to establish a working fresh install to explore the 3.6 path.

What should I do to ever downgrade to 3.4 or make the 3.6 version run? A solution which drop the db is fine, this is a development station, I have a backup ready to be imported, although it's a dump from a mongo 3.4 instance.

Best Answer

So, I finally found a solution which seems to work when going on the move forward to mongodb 3.6 path.

Warning: remember that in this case a proper backup was already available, this solution remove all existing data that were stored in mongodb so far. Don't do that if no backup of the database is available.

sudo dnf remove --all mongodb mongodb-server mongo-tools                                                                   
# seems like it's not enough to remove the data too so let's do it manually
cd /var/lib/
sudo mv mongodb mongodb-3.4 # or `rm -fr mongodb` directly if you don't mind
sudo dnf install mongodb mongodb-server mongo-tools
sudo systemctl start mongod
mongorestore --drop --gzip --archive="$path_to_mongodb_backup.gz" 

And that's it.