What’s new in Apache CouchDB 0.11 — Part Three: New Features in Replication

Hey, this is part three of my small series on new features in CouchDB 0.11. Don’t miss part one and part two.

Without a doubt, replication is CouchDB’s coolest feature. It allows you to synchronise any two databases, local or remote. There is no role-distinction between databases, no master-slave limitations, everybody is a master (if you want master-slave, simply don’t write to one master and thus make it a slave).

This allows you to build a replication infrastructure that fits your application and deployment needs best: two offices with an ocean in between, no problem; large server cluster in one or more data centres, no problem. And anything in between really.

Replication is not new, it has been baked into CouchDB from the beginning. Today, I’ll show you some of the nifty features we added to the 0.11 replicator to make your life a little easier.

To recap, here is how you trigger replication from CouchDB running on your local machine to synchronise a local database with a remote database:

curl -X POST http://127.0.0.1:5984/_replicate \
-d '{"source":"database", \
     "target":"http://example.com:5984/database"}

Note: The backslash \ means “keep reading the command on the next line” to make things more readable and so you can copy and paste the commands into your command line.

If you want to read up on CouchDB replication, check out the chapters Replication and Conflict Management from our book, CouchDB: The Definitive Guide.

Implicitly Create Target Database

This is a small yet useful tip. Prior to 0.11, if you wanted to replicate, the target database would have to exist. This can be a mild annoyance at times, so we added a replicator option to implicitly create target databases, if they don’t exist.

Using it is easy, here’s the curl command:

curl -X POST http://127.0.0.1:5984/_replicate \
-d '{"source":"database", \
     "target":"http://example.com:5984/database", \
     "create_target":true}'

Et Voilà.

I know, not too impressive, but boy do I like it :) Moving on.

Replicate Documents by Id

Replication works from a source to a target database. By default, CouchDB finds all the documents that exist on the source database that are not on the target database and then sends these to the target database. Pretty simple and powerful.

With 0.11, instead of replicating the difference between source and target, you can specify a list of document ids to be replicated from the source to the target. This is useful when you only want a subset of your documents to be replicated, say only design documents.

Here is how it works:

curl -X POST http://127.0.0.1:5984/_replicate \
-d '{"source":"database", \
    "target":"http://example.com:5984/database", \
    "doc_ids":["_design%2fapp", "support"]}'

Pretty simple, again. The above command replicates only the two documents with the ids "_design/app" and "support".

Note that for design documents, you need to URL escape the / to %2f. (1.0 will likely allow you to just use the slash / verbatim).

I hope you like this one as much as I do. It makes selectively deploying documents a lot easier. If you are building CouchApps, this one is a real time-saver.

Filters

Moving on to replication filters. I wanted this for a long time and I am really excited that CouchDB supports them now. There is a bunch of interesting scenarios where they are useful. I’ll get to the scenarios, but first, let’s have a look at how replication filters work.

Replication uses CouchDB’s _changes feed to figure out what’s new in the source database that needs to get shipped to the target. I wrote about the _changes feed in detail in our book CouchDB: The Definitive Guide.

Version 0.10 added a continuous: true option that would start replication and keep it running so that documents added to the source database would immediately be replicated to the target database. It makes use of the feed=continuous option for the _changes feed. The book link has all the details.

In 0.11 you can also specify a filter for the _changes feed:

curl -X POST http://127.0.0.1:5984/_replicate \
-d '{"source":"database", \
     "target":"http://example.com:5984/database", \
     "filter":"example/filtername"}'

Consider this filter function:

function(doc, req) {
  if(doc._id.charAt(0) == "a") {
    return true;
  }
  return false;
}

It returns true for all documents that have an id that starts with a. Bear with me for the stupid example to explain how this works.

A filter function gets passed each document that is to be sent down the _changes feed. CouchDB expects a boolean return value. Returning false makes CouchDB not send the document change, true will send it. Pretty simple.

To tell CouchDB what you want to call your filter function, it must be defined in a design document (Update: …in your source database):

{
  "_id": "_design/example",
  "filters": {
    "filtername":"function(doc, req) { ... }"
  }
}

The req parameter includes all the HTTP request information (headers, method, query parameters etc.) of the request that calls the _changes feed.

Say you want to replicate all documents with a specific name in a field called username, but don’t want to hardcode the name in the filter function or create a filter function for each name:

function(doc, req) {
  // make sure we don't access any fields that 
  // might be undefined
  if(!doc.username) {
    return false;
  }

  if(!req.query.name) {
    throw("Please provide a query parameter `name`.");
  }

  // else

  if(doc.username == req.query.name) {
    // the query parameter `name` matches 
    // the corresponding field in `doc`
    return true;
  }

  // by default, don't send anything
  return false;
}

Now use this request to trigger replication:

curl -X POST http://127.0.0.1:5984/_replicate \
-d '{"source":"database", \
     "target":"http://example.com:5984/database", \
     "filter":"example/filtername", \
     "query_params": { \
       "name": "Pete" \
     }}'

CouchDB will send this request to the _changes feed:

GET /database/_changes?name=Pete

Inside the filter function, you can access the the value of name through req.query.name. The rest is deciding whether to return true or false.

After replication finishes, the target database will only have new documents that have a field username with the value of "Pete". Mixing filters and continuous replication just works as you would expect it.

Use Cases

This sums up the mechanics, but what can we use this for?

Use Case: Replicating Inboxes or “Idle users are free”

Say you have a huge database with messages that users send to each other. Say you want to distribute all messages to their recipient’s inbox which is just another database that holds all the users data.

Having a database per user is very convenient if you need to logically separate data. You can specify individually who can read or write to them, you can move them around individually in your infrastructure; a busy user could be moved over to an idle cluster easily. All the data for a user is contained in a single file on disk. Things get a lot more tricky if you have one major database for everybody.

CouchDB has been tested with over a million users per instance and is working fine. The biggest production database we know of has about half a million users with one or more databases each.

However, you want to avoid copying messages when you don’t have to. Users shouldn’t get messages delivered when they are not active. When your application detects a user logging into the system, it can trigger filtered replication to update the user’s inbox. If a user never logs in, you don’t duplicate any data. I like to call this “Idle users are free.”

Inbox Delivery

Sounds good? Let’s do it. Here is an example message:

{
  "from": "The Hungarian",
  "text": "My hovercraft is full of eels.",
  "recipients": [
    "tobacconist",
    "constable",
    "judge"
  ]
}

Consider this filter function:

function(message, req) {
  // require a valid request user
  if(!req.userCtx.name) {
    throw("Unauthorized!");
  }

  // only look at messages that have recipients
  if(!message.recipients) {
    return false;
  }

  if(message.recipients.indexOf(req.userCtx.name) !== -1) {
    // our user is in the recipients list
    return true;
  }

  // default
  return false;
}

If the request field userCtx.name matches a recipient in a message, replicate it.

What is this userCtx? When you make an authenticated request against CouchDB, either using HTTP basic auth, secure cookie auth or OAuth, CouchDB will verify the user’s credentials. If they match a CouchDB user, it populates the req.userCtx object with information about the user.

If we are in a filter function and req.userCtx.name is defined, we can be sure that this is a valid user that has sent the correct login credentials with the request. That way we can guarantee that users can only see messages delivered to their inbox that they are supposed to see.

As an added bonus, delivering a message is as easy as creating a new message document in the main database with a recipients list.

Use Case: Splitting Cluster Nodes

This is bit of an advanced one. Say you have a bunch of CouchDB nodes that all hold a part of a larger database. You might be using CouchDB-Lounge for that.

CouchDB-Lounge defines a consistent hashing ring over all documents. Each node or shard gets a subset of all documents assigned to it. If you have 4 machines, each get 1/4th of the documents.

Lounge uses a HTTP proxy based on nginx to distribute queries among the nodes inside the cluster. To the outside, Lounge looks like a single CouchDB with a lot of power.

This is great for distributing load and adding write- and storage-capacity in your setup. But how does this grow? Just remember the last paragraph: to the outside, Lounge looks like a regular CouchDB instance. Just replace each of your nodes with another setup of CouchDB-Lounge. Problem solved. I like to call this fractal scaling.

Fractal Scaling

This requires us to prepare another, say 4, nodes with 1/4 of the previous node’s documents (or 1/16th of the total number of documents (look at me, the math whizz)). How can we use replication filters to split all documents and distribute them to the new nodes?

Splitting a Node

Let’s say you are using UUIDs to identify your documents. This gives you a reasonably random distribution, i.e. all nodes should get an equal share of all documents. Say all documents that have ids starting with 0-3 are assigned to node 0, 4-7 to node 1, 8-B (B hex = 11 dec) to node 2 and C-F to node 3.

Now let’s say we want to split all nodes to add capacity. We start with one node to keep additional load on the system low. Once we know how to split one node, we can apply the same procedure to all the others in the same way.

Let’s start with node 0. The new nodes 0-0, 0-1, 0-2 and 0-3 will be responsible for the id prefixes 0, 1, 2 and 3 respectively. We can use this filter function to make the split:

function(doc, req) {
  // we require the client to specify the 
  // key-range he is interested in
  if(!req.query.prefix) {
    throw("Please specify `?prefix=whatever`");
  }

  var prefix = req.query.prefix;
  if(doc._id.substr(-prefix.length) == prefix) {
    // the doc id matches the supplied prefix
    return true;
  }

  // default
  return false;
}

If you now add the correct query parameter to the filtered replication request, you’ll only get the docs you’re interested in:

curl -X POST http://127.0.0.1:5984/_replicate \
-d '{"source":"database", \
     "target":"http://example.com:5984/database", \
     "filter":"example/filtername", \
     "query_params": { \
       "prefix": "0" \
     }}'

Nodes 0-1, 0-2 and 0-3 use "1", "2" and "3".

That’s it. Rinse and repeat.

Use Case: DesktopCouch

About a year ago Canonical, makers of the insanely popular (and good!) Linux distribution Ubuntu (you know that, of course) approached the CouchDB team and asked if it would be suitable to power their idea for sharing personal data between desktop, mobile and cloud computers.

Turns out CouchDB not only fits, but is very well suited. Their idea was to extend all the applications that store personal data like address book entries, bookmarks and notes (to begin with) and have these applications store into CouchDB instad of the filesystem or SQLite. Users should be able to synchronise their personal data between multiple machines (home, mobile, work). CouchDB replication allows them to do that.

On top of that, Canonical offers a cloud backup and distribution service called UbuntuOne that gives every registered user 2GB of free storage for their personal data. More space is available to paying customers. CouchDB powers both the local synchronisation and the cloud synchronization easily.

If you or your company have sensitive data they don’t want to share with UbuntuOne, they can synchronise their machines locally. Users who trust Canonical (they are very good with security, I trust them) can rely on a remote backup server that keeps all their data safe in case their computer breaks or gets lost.

Sounds useful? Damn right. With Ubuntu 9.10 (Karmic Koala) this vision is available to all users of Ubuntu. Canonical provides plug-ins for Firefox, Evolution, Tomboy and other open source applications to use CouchDB as a storage for personal data that users may want to have synchronised.

It’s worth noting that this is all opt-in, nobody is forced to use the infrastructure if they don’t want to.

Underlying all the implementation work sits a project called DesktopCouch that is (so far) a joint effort of Mozilla and Canonical to standardise the necessary bits to make CouchDB useful for personal data sync services. The project is open to anyone and encourages other platforms like Windows, Mac OS X and other Linux distributions to implement the open standard and allow for seamless synchronisation.

DesktopCouch gives the user full control over their data. When Canonical asked us how they can lock down CouchDB so users wouldn’t tinker with it we simply asked “What’s wrong with tinkering?”. Canonical agreed that this is a good thing and now every DesktopCouch user is able to do whatever with the data stored in their local CouchDB instance.

They could be writing specialised CouchApps to access data in some innovative way through the browser or build native applications with novel user interfaces without having to worry about data interchange.

While DesktopCouch originated from the Ubuntu desktop system, it is platform & vendor agnostic. Both Canonical and Mozilla actively contribute and welcome everybody else, too. If you are running a different Linux distribution, or even Mac OS X or Windows, there’s no reason why you shouldn’t be able to use the same tools and standards to manage your personal data.

If you are interested in bringing DesktopCouch to your platform of choice, get in touch and I’ll hook you up with the right people.

Use Case: Need-to-know-based Data Sharing

Everybody likes secret agent titles. Our last filtered replication scenario involves multiple parties with varying trust levels. CouchDB’s filtered replication can ensure the right information ending up in the right hands, but never in the wrong ones.

The cast: Sears, FedEx, UPS and the IRS. The scenario: hypothetical but plausible A little bit more complicated senario is in production at www.assaydepot.com, the outlined scenario here is simplified to make it easier to explain.

Enough with the preface, here’s the scenario: Sears uses multiple carriers (FedEx, UPS) for shipping stocked items around. They have a central database with all the items they need to ship. The carriers have a local database that replicate from Sears’ central database. They’ll only ever see the items they are supposed to ship never any of the items a competitors is shipping.

Each carrier then updates the documents related to shipped items with their shipping status and all the other information that is relevant. Sears then can replicate the carriers’ databases back into their central database to get an aggregated view of all items that are in shipment with any carrier.

That way Sears is in full control over all the shipping information. They can optimise the entire system based on each carriers performance and find out the optional way to organise shipping goods without having to disclose this information to the carriers.

Now, in order to qualify for some tax cuts on shipping, Sears has to document and show to the IRS that their shipping system operates within a certain limit of metrics defined by the IRS. Sears can provide the IRS with a separate filter to replicate out all documents that document the shipping history of the last month along with the MapReduce View definitions that run the calculations to if Sears is eligible for the tax cut.

The IRS can provide their own MapReduce views to verify the Sears results. Neither the IRS nor Sears have to make public any data or procedures that or not strictly necessary for determining Sears’ tax status.

Everybody wins and the process can be automated and is easy to audit.

Wrapping Up

Is your head not spinning yet? Mine is, but can you come up with more scenarios where CouchDB replication can save the day by being awesome? Write it up in your blog and we can sum up all approaches in a follow-up blog post here.

Hope you enjoyed this one!

Love, Jan.