kchodorow

MongoDB's replica sets are very similar to a normal master-slave setup, except they are self-monitoring: if the master goes down, the slaves will band together to promote a new master automatically.

It's easy to try this out and fun to watch the automatic failover (or perhaps I'm easily amused). You'll need to download MongoDB and the MongoDB Perl driver (cpan MongoDB). Once you unzip the MongoDB binaries, you're ready to spin up a replica set (there's no install necessary).

Change to the MongoDB directory you downloaded and start up three MongoDB servers. Each database server needs:

1. Somewhere to put its data (a directory).
2. A unique port to listen on (it's a server).
3. The name of the replica set it's a member of. Each replica set needs a unique name, so for this example I'll use mongoMonger.

In the "real world" (production), you'd have each member of the set on a different machine, but I'm setting them up locally in case you only have one machine to work with.

To start up the set, run:

$ # create the data directories
$ mkdir -p ~/dbs/monger1 ~/dbs/monger2 ~/dbs/monger3
$ # start the servers
$ bin/mongod --dbpath ~/dbs/monger1 --port 27017 --replSet mongoMonger
$ bin/mongod --dbpath ~/dbs/monger2 --port 27018 --replSet mongoMonger
$ bin/mongod --dbpath ~/dbs/monger3 --port 27019 --replSet mongoMonger

Run each of the three ./mongod commands in its own terminal, as they are ongoing processes and we want to watch the output.

Now we need to tell the members of the replica set about each other. We can use the MongoDB shell for this. From the directory you're in, run bin/mongo (no "d") to start the shell. At the prompt, send the set's members to the rs.initiate function:

> rs.initiate({"_id" : "mongoMonger", "members" : [
... {"_id" : 0, "host" : "localhost:27017"},
... {"_id" : 1, "host" : "localhost:27018"},
... {"_id" : 2, "host" : "localhost:27019"}]})

Press enter and you should see the message "Config now saved locally. Should come online in about a minute." You're replica set is ready to go!

Now, to try out the automatic failover, copy this Perl script to file:

use strict;
use warnings;
use MongoDB;

my $m = MongoDB::Connection->new(host => "mongodb://localhost:27017", find_master => 1);

my $c = $m->foo->bar;

while (1) {                                                                                                       

   eval {

       $c->find_one();

   };

   if ($@) {

       print $@;

   }

   else {

       if ($m->_master){

           print "connected to: ".$m->_master->{host}."\n";

       }

       else {

           print "no master\n";

       }

   }

   sleep 1;

}

This script queries the database once every second. At the beginning of the script, we specify that we want to connect to a member of the replica set and that we want the driver to automatically detect the master (find_master => 1).

If you run this script, you'll see output that looks something like:

connected to: mongodb://localhost:27017
connected to: mongodb://localhost:27017
connected to: mongodb://localhost:27017

This is the current master for the replica set. If you go to whichever terminal is running the server listed (localhost:27017 in this case) and kill the mongod process, you'll see errors for a few seconds and then the replica set will elect a new master:

no master
can't get db response, not connected at /usr/local/lib/perl5/site_perl/5.12.0/i686-linux-thread-multi/MongoDB/Cursor.pm line 230.
can't get db response, not connected at /usr/local/lib/perl5/site_perl/5.12.0/i686-linux-thread-multi/MongoDB/Cursor.pm line 230.
connected to: mongodb://localhost:27018
connected to: mongodb://localhost:27018
connected to: mongodb://localhost:27018

As you can see, the replica set auto-heals and the driver figures out which server became the new master.

You can now bring back up the mongod that you just killed, wait a few seconds, and kill the new master. You can do this ad infinitum, killing masters and bringing them back to life and watching the driver fail over.

In a more practical sense, this means that your application can have terrific availability and you don't have to get paged to do it yourself at 3am.

Further Reading

• Official MongoDB documentation on replica sets
• A blog post I wrote on how replica sets work (based on a question someone asked at YAPC::NA, actually)