Are you running from the 10gen repository or from the default Debian/Ubuntu repo? I recommend using the official 10gen repository.
Check this link out - [10gen MongoDB How-To Install on Ubuntu:] http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/. It is best to uninstall the previous mongodb installation prior to this change, which will also require you to modify your repository source (in /etc/apt/source.list) but this is also detailed in the link above. I recommend using upstart over sysvinit and the processed is outlined in the How-To.
With the 10gen Ubuntu set-up, the configuration file is in /etc/mongodb.conf.
There are several ways that you can you can run a separate mongod process, one would simply be running it from the cli -
sudo -u mongodb mongod --dbpath /var/tmp/mongotest --logpath /var/tmp/mongotest_log --port 3001 &
which will produce (using 'ps')
mongodb 2210 3.3 1.5 259012 15300 pts/0 Dl 11:48 0:00 mongod --dbpath /var/tmp/mongotest --logpath /var/tmp/mongotest_log --port 3001
and you can connect via the cli -> mongod --port 3001
Another way is creating a copy of /etc/mongodb.conf -
sudo cp /etc/mongodb.conf /etc/mongodbnew.conf and editing the following lines in the new file -
# mongodbnew.conf
dbpath=/var/lib/mongodbnew
#where to log
logpath=/var/log/mongodb/mongodbnew.log
port = 27018
I have changed the dbpath (where the mongodb files are stored) from /var/lib/mongodb to /var/lib/mongodbnew; the log path has changed from mongodb.log to mongodbnew.log and the port has changed from the default of 27017 to 27018 (you will also have to remove the # to uncomment the line).
I also changed the top line to reflect the new name of this configuration file.
You will also have to create the data directory because it won't exist and without it, the mongod process won't start and ensure that the owner and group is mongodb -
sudo mkdir /var/lib/mongodbnew
sudo chown -R mongodb:mongodb mongodbnew/
Additionally, create and change the persmissions of your log file -
sudo touch /var/log/mongodb/mongodbnew.log && sudo chown mongodb:mongodb /var/log/mongodb/mongodbnew.log
To start your new mongod process (in the background) from the cli, type
sudo -u mongodb /usr/bin/mongod --config /etc/mongodbnew.conf &
which send the following to the screen -
"all output going to: /var/log/mongodb/mongodbnew.log"
Check that the mongod processes are running with "ps" -
sysadmin@ubuntu:/var/lib$ ps auwx | grep mongo | egrep -v 'grep|sudo'
mongodb 921 11.3 1.5 627672 15608 ? Ssl 10:56 4:30 /usr/bin/mongod --config /etc/mongodb.conf
mongodb 2137 7.8 1.5 627672 15880 pts/0 Sl 11:36 0:00 /usr/bin/mongod --config /etc/mongodbnew.conf
To verify that you can connect, run the Mongo shell -
$ mongo --port 27018
MongoDB shell version: 2.0.5
connecting to: 127.0.0.1:27018/test
>
With lsof, you should now see your two mongod processes, the original bound to 27017 and the new one bound to 27018.
$ sudo lsof -i :27017
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mongod 921 mongodb 6u IPv4 9066 0t0 TCP *:27017 (LISTEN)
sysadmin@ubuntu:/var/lib$ sudo lsof -i :27018
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mongod 2137 mongodb 6r IPv4 11923 0t0 TCP *:27018 (LISTEN)
To ensure that the two processes run at start-up, copy the original init configuration file -
sudo cp /etc/init/mongodb.conf /etc/init/mongodbnew.conf
which should result in two files such as -
$ ls -lart /etc/init/mongo*
-rw-r--r-- 1 root root 536 May 8 15:51 /etc/init/mongodb.conf
-rw-r--r-- 1 root root 554 Jun 11 11:43 /etc/init/mongodbnew.conf
Edit the new mongodbnew.conf file using vi or whatever you prefer to look like this -
## Ubuntu upstart file at /etc/init/mongodbnew.conf
limit nofile 20000 20000
kill timeout 300 # wait 300s between SIGTERM and SIGKILL.
pre-start script
mkdir -p /var/lib/mongodbnew/
mkdir -p /var/log/mongodbnew/
end script
start on runlevel [2345]
stop on runlevel [06]
script
ENABLE_MONGODB="yes"
if [ -f /etc/default/mongodbnew ]; then . /etc/default/mongodbnew; fi
if [ "x$ENABLE_MONGODB" = "xyes" ]; then exec start-stop-daemon --start --quiet --chuid mongodb --exec /usr/bin/mongod -- --config /etc/mongodbnew.conf; fi
end script
I have tested this on Ubuntu 12.04 and it appears to work well, including after a reboot. I haven't run it in production obviously. Apologies, if there's any typo's above but there's a lot of information and I may have missed something.
Finally, here's a link on Replica Sets that might help you as it has come examples on starting multiple mongod instances from the cli - http://www.mongodb.org/display/DOCS/Replica+Set+Tutorial.
Restoring config servers, particularly if you have had some sort of catastrophic event is tricky, but not impossible. But, before we go any further, a big bold caveat:
BACK UP EVERYTHING
That means taking a back up of all three config servers. I am going to give you some advice, and it is generally correct, but please, please take a back up of every current config server instance before you overwrite/replace anything
As a quick explanation, config servers are not configured as a replica set - each config server instance is supposed to be identical (at least for all the collections that matter) to the others. Hence, any healthy config server can be used to replace a non-healthy config server and you can then follow the tutorial you mentioned to get back to a good config.
The key to recovery is to identify the healthy config server and then use that to replace the others - you then end up with 3 identical config servers.
There is more than one way to do this, they basically fall into three categories:
1) Use the error message
The error message that is printed out actually lets you know which config server it believes is health, though that is not obvious from the messaging. Here's how to read it generically:
ERROR: config servers not in sync! config servers <healthy-server> and <out-of-sync-server> differ
Basically the first one in the list is the healthy one, in your case that would be mongocfg1.testing.com:27000. That is our first candidate for a healthy config database.
2) Use dbhash to compare all three and pick the ones that agree
On each config server switch to the config database using use config, run db.runCommand("dbhash") and compare the hashes for the collections below:
- chunks
- databases
- settings
- shards
- version
You are looking for two servers that agree, and using that as the basis to determine that the version of the config database on those hosts is basically trustworthy and should be used to seed the rest.
3. Manually inspect the collections in the config database
Finally, take a look at the config database, and pay attention to the collections listed in the second option above. This is a straight judgement call based on your familiarity with your data.
Hopefully all three methods point you at the same host (or hosts). That config server should be used to seed the other two (after you have taken backups so you can go back). That is basically your best bet. Should that fail, then you may want to try one of the other versions (from the backups) - always making sure that when you start them, all three are identical.
Finally, always ensure that all mongos processes are using the same config server string, and that all 3 servers are always listed in the same order on every process - not doing so across all mongos processes can lead to (very) odd results.
Best Answer
It seems as though it was reading the config file but reading it in a different way that it was ignoring a certain directive. I tried setting smallfiles to true which worked when starting manually. But in service mode I got it to work by disabling journaling. Not sure if this is a bug.