Managed Services

SETTING UP A MONGO DB REPLICA SET

MongoDB is arguably the leading NoSQL database available today, it makes it super easy to start-off on projects without having to worry about table schemas, while delivering extremely fast performance and lots of useful features.

REPLICA SETS :

It is a group of Mongo DB servers operating in a primary/secondary fail over manner. At any point there can only be one primary member within the replica set, however, you can have as many secondaries as you want. All secondaries actively replicate data off of the current primary member so that if it fails, one of them will be able to take over quite seamlessly as the new primary.

The more secondaries you have, and the more spread out they are over availability zones or regions, the less chance your cluster will ever experience downtime.

REPLICA SET MEMBERS :

The most minimal replica set setup must have at least three healthy members to operate. One member will serve as the primary, another as the secondary, and the third as an arbiter.

Arbiters are members that participate in elections in order to break ties and do not actually replicate any data. If a replica set has an even number of members, we must add an arbiter member to act as a tie-breaker, otherwise, when the primary member fails or steps down, a new primary will not be elected.

You can also avoid an arbiter by simply adding another secondary instead. All you really need is an odd number of members for elections to be held properly. However, an extra secondary will cost you more than an arbiter.

 

NOTE : The below installation steps are for a MONGO replica set with 1 primary and 2 secondaries.

CONFIGURATION :

Select the desirable instance type on AWS along with the required storage and the necessary configurations and get started with the Mongo DB replica sets configuration.

Once the ec2-instance is set up on AWS, log into the server and perform the below steps.

SERVER HARDENING :

  1. Increase OS Limits

MongoDB needs to be able to create file descriptors when clients connect and spawn a large number of processes in order to operate effectively. The default file and process limits shipped with the OS (in this case Red Hat Enterprise Linux 7) are not applicable for MongoDB.

Modify them by editing the limits.conf file:

vi /etc/security/limits.conf

Add the following lines to the end of the file:

* soft nofile 64000
* hard nofile 64000
* soft nproc 32000
* hard nproc 32000

Next, create a file called 90-nproc.conf in /etc/security/limits.d/:

vi /etc/security/limits.d/90-nproc.conf
Paste the following lines into the file:

* soft nproc 32000
* hard nproc 32000

2. Disable Transparent Huge Pages

Transparent Huge Pages (THP) is a Linux memory management system that reduces the overhead of Translation Lookaside Buffer (TLB) lookups on machines with large amounts of memory by using larger memory pages.

However, database workloads often perform poorly with THP, because they tend to have sparse rather than contiguous memory access patterns. You should disable THP to ensure best performance with MongoDB.

Run the following commands to create an init script that will automatically disable THP on system boot:

vi /etc/init.d/disable-transparent-hugepages
Paste the following inside it:

#!/bin/sh
### BEGIN INIT INFO
# Provides: disable-transparent-hugepages
# Required-Start: $local_fs
# Required-Stop:
# X-Start-Before: mongod mongodb-mms-automation-agent
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description: Disable Linux transparent huge pages, to improve
# database performance.
### END INIT INFO

case $1 in
start)
if [ -d /sys/kernel/mm/transparent_hugepage ]; then
thp_path=/sys/kernel/mm/transparent_hugepage
elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
thp_path=/sys/kernel/mm/redhat_transparent_hugepage
else
return 0
fi

echo ‘never’ > ${thp_path}/enabled
echo ‘never’ > ${thp_path}/defrag

unset thp_path
;;
esac
Make it executable:

chmod 755 /etc/init.d/disable-transparent-hugepages
Set it to start automatically on boot:

chkconfig disable-transparent-hugepages on
To confirm whether this has been enabled or not use : chkconfig –list
The output of this command should have the below entry ;
disable-transparent-hugepages 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Reboot the instance for the changes to reflect by using ‘reboot’ command.

Verification :

After rebooting, you can check whether the changes that you have made are being reflected or not :

  1. Check that the OS limits have been increased by running:

ulimit -u # max number of processes
ulimit -n # max number of open file descriptors
The first command should output 32000, the second 64000.

2. Check whether the Transparent Huge Pages feature was disabled successfully by issuing the following commands:

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag
For both commands, the correct output resembles:

always madvise [never]

NOTE : Please repeat  the server hardening steps on all the servers on which your Mongo DB’s will be running.

(OR)

An even better idea is to have the whole thing set-up on a single server  (along with the mongo replica set-up) and then before starting the mongo DB you could take the snapshot of the instance and use the snapshot for the other instances. This will save you time and repetitive performance of the same steps could be avoided.

INSTALLATION STEPS :

Download mongo at the desired location say /opt/mongodb (mongodb is the folder that has been created in opt):

wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.2.7.tgz

tar -xzvf mongodb-linux-x86_64-3.2.7.tgz
all the files will be extracted to this folder
mv mongodb-linux-x86_64-3.2.7/* /opt/mongodb

create folders data and logs inside mongodb folder
mkdir -p /opt/mongodb/data/db
mkdir -p /opt/mongodb/logs
touch /opt/mongodb/logs/mongodb.log (creates an empty file)
create file mongod.conf under /opt/mongodb
vim mongod.conf    — After creating this, add the below steps to the file.
####add below####
systemLog:
destination: file
path: “/opt/mongodb/logs/mongodb.log”
logAppend: true
quiet: false

processManagement:
fork: true

net:
bindIp: 0.0.0.0
port: 27017

storage:
dbPath: /opt/mongodb/data/db
indexBuildRetry: true
journal:
enabled: true
commitIntervalMs: 100
directoryPerDB: true

replication:
oplogSizeMB: 2048
replSetName: rs

######################################

Newer version of mongo db accept ‘.conf’ as an extension to the mongo conf file and also the file should be in proper format.
Make sure bindIp is 0.0.0.0 , because 0.0.0.0 will bind all the network interfaces available. If 127.0.0.1 is given then its only bound to the loopback interface (which means the server can listen to itself but the other mongos residing outside the server cannot connect to it)
Further explanation to this could be found at : http://stackoverflow.com/questions/17588876/mongodb-conf-bind-ip-127-0-0-1-does-not-work-but-0-0-0-0-works
Give any name for replica set (rs as in screenshot)

Note : Do not start Mongo DB right away. Please follow the below steps for starting mongo.
Starting the Mongo :

This section talks about starting the Mongo DBs in an order.

Based on the architecture start your primary mongo and secondary mongos accordingly.

Since all the configurations are updated in mongod.conf
Start the primary mongo first with the following command (from the bin folder)
./mongod –config ../mongod.conf

Check the logs.

Then start your secondary mongos later by logging into their instances and running the same command as above. (./mongod –config ../mongod.conf)

************************************************************
After the secondary mongos are also started without any errors then login to one of the mongo shells which you want to work as primary.
Run these on the primary mongo shell : (From the bin folder)
./mongo
rs.initiate() — This command will initiate adding the present instance as primary to the replica set.
rs.conf() — This command is used to validate whether the primary got added to replica set.
rs.add(“10.145.xx.xx:27017”) — Similarly add all your replica set members(make sure all the replica set members are running as well)
validate the same with rs.conf — This lists all the replica set members.

In case, if the primary mongo goes down , then either of replica set members will be coming up as primary. And when the original primary is recovered it becomes a secondary member.
So ideally there is no concept of having one single primary mongo, this is why we started one mongo with rs.initiate() which will be adding the present replica set member itself as a primary.

To check which mongo is primary type use : db.isMAster()

Follow the below steps if you want to delete a DB or shut down the mongo :

######delete a database#####
./mongo
use admin
use <name of the database>
db.dropDatabase()

#######shutdown mongo#####
./mongo
use admin
db.shutdownServer()

About The Author

Leave a Reply

*