MongoDB won’t start with access control turned on

mongodbmongodb-3.0

Hello I would appreciate any suggestions, I have a 3 member MongoDB replica set setup on CentOS I'm trying to configure wit access control:
node1,
node2,
arbiter,

When I do not configure security with access control to test each instance it works successfully. I didn't get as far to test replication, although I have tested in the past with no access control however I need it implemented.

I get an error and mongodb wont start with access control on.
I've granted mongod account permissions "/srv/mongodb/mongodb-keyfile.key"
set chmod 600 as instructed by the docs.

The get the following messages in the logs but doesn't start:

2016-03-11T09:42:25.032-0800 I CONTROL  [main] ***** SERVER RESTARTED *****
2016-03-11T09:42:25.094-0800 I CONTROL  [initandlisten] MongoDB starting : pid=8729 port=27000 dbpath=/var/lib/mongo/arb 64-bit host=arb
2016-03-11T09:42:25.094-0800 I CONTROL  [initandlisten] db version v3.2.3
2016-03-11T09:42:25.094-0800 I CONTROL  [initandlisten] git version: b326ba837cf6f49d65c2f85e1b70f6f31ece7937
2016-03-11T09:42:25.094-0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-03-11T09:42:25.094-0800 I CONTROL  [initandlisten] allocator: tcmalloc
2016-03-11T09:42:25.094-0800 I CONTROL  [initandlisten] modules: none
2016-03-11T09:42:25.094-0800 I CONTROL  [initandlisten] build environment:
2016-03-11T09:42:25.094-0800 I CONTROL  [initandlisten]     distmod: rhel62
2016-03-11T09:42:25.094-0800 I CONTROL  [initandlisten]     distarch: x86_64
2016-03-11T09:42:25.094-0800 I CONTROL  [initandlisten]     target_arch: x86_64
2016-03-11T09:42:25.095-0800 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1,192.222.21.3", port: 27000 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo/arb", journal: { enabled: false }, mmapv1: { smallFiles: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2016-03-11T09:42:25.166-0800 I -        [initandlisten] Detected data files in /var/lib/mongo/arb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-03-11T09:42:25.186-0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2016-03-11T09:42:25.677-0800 W STORAGE  [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger
2016-03-11T09:42:25.677-0800 I CONTROL  [initandlisten]
2016-03-11T09:42:25.677-0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2016-03-11T09:42:25.677-0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2016-03-11T09:42:25.677-0800 I CONTROL  [initandlisten]
2016-03-11T09:42:25.677-0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-03-11T09:42:25.677-0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2016-03-11T09:42:25.677-0800 I CONTROL  [initandlisten]
2016-03-11T09:42:25.677-0800 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 1024 processes, 65535 files. Number of processes should be at least 32767.5 : 0.5 times number of files.
2016-03-11T09:42:25.677-0800 I CONTROL  [initandlisten]
2016-03-11T09:42:25.741-0800 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/arb/diagnostic.data'
2016-03-11T09:42:25.742-0800 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-03-11T09:42:25.776-0800 I NETWORK  [initandlisten] waiting for connections on port 27808
2016-03-11T09:42:29.954-0800 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2016-03-11T09:42:29.954-0800 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2016-03-11T09:42:29.955-0800 I CONTROL  [signalProcessingThread] now exiting
2016-03-11T09:42:29.955-0800 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2016-03-11T09:42:29.955-0800 I NETWORK  [signalProcessingThread] closing listening socket: 6
2016-03-11T09:42:29.955-0800 I NETWORK  [signalProcessingThread] closing listening socket: 7
2016-03-11T09:42:29.955-0800 I NETWORK  [signalProcessingThread] closing listening socket: 8
2016-03-11T09:42:29.955-0800 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27000.sock
2016-03-11T09:42:29.955-0800 I NETWORK  [signalProcessingThread] shutdown: going to flush diaglog...
2016-03-11T09:42:29.955-0800 I NETWORK  [signalProcessingThread] shutdown: going to close sockets...
2016-03-11T09:42:29.955-0800 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2016-03-11T09:42:30.014-0800 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2016-03-11T09:42:30.014-0800 I CONTROL  [signalProcessingThread] dbexit:  rc: 0

I have followed steps up to the point confirming that all 3 hosts can "start up" as separate MongoDB instances
https://docs.mongodb.org/manual/tutorial/enable-internal-authentication/#deploy-repl-set-with-auth

# mongod.conf
# Arbiter

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /var/lib/mongo/arb
  smallFiles: true
  journal:
    enabled: false
#  engine:
#  mmapv1:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile

# network interfaces
net:
  port: 27000
  bindIp: 127.0.0.1,192.222.21.3  # Listen to local interface only, comment to listen on all interfaces. I've changed the IP to mask it

# when the follow 3 lines below are commented out it works
security:
 keyFile:  "/srv/mongodb/mongodb-keyfile.key"
 authorization: true

#operationProfiling:

# commented out to test access control first
#replication:
#  oplogSizeMB: 1024
# replSetName: "rs1"

#sharding:

## Enterprise-Only Options

#auditLog:

#snmp:

Best Answer

I solved my issue by deleting and moving the section below and removing quotes, it's strange solved it. There must have been some hidden characters. Lesson learned delete you config and start over maybe.

# when the follow 3 lines below are commented out it works
security:
  keyFile:  /srv/mongodb/mongodb-keyfile.key
  authorization: true