Skip to content

Migrating Replica Set Online

Jump to bottom
doluiscontreras edited this page Jul 9, 2015 · 8 revisions

Migrating from MongoDB: Replica set (online)

This guide explains how to do an online data migration from MongoDB to TokuMX, for a replica set, converting the existing data in MongoDB to TokuMX.

For other migration strategies, start with Migrating from MongoDB and Migrating from MongoDB: Replica sets.

You will need to prepare by provisioning a number of extra machines where you will build your TokuMX replica set.

The online migration of a replica set is four phases:

  1. Backup
  2. Import
  3. Catchup
  4. Switch

Backup

If you already have a backup with a known OpTime, you can skip the first steps that take a backup of a secondary.

  1. Shut down one of the MongoDB secondaries.

  2. Take a backup of that secondary with mongodump.

    $ sudo mongodump --dbpath /var/lib/mongodb --out /var/lib/mongodb.backup

- MongoDB latest version -

$ sudo mongodump --out /var/lib/mongodb.backup

  3. While the secondary is down, connect to the primary and find out the OpTime that the secondary was at when it was shut down. Look for the optime for the machine in the members array that is "(not reachable/healthy)".

    rs:PRIMARY> rs.status()
{
        "set" : "rs",
        "date" : ISODate("2013-07-13T18:19:04Z"),
        "myState" : 1,
        "members" : [
                ...,
                {
                        "_id" : 2,
                        "name" : "cavil:20002",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "t" : 1373658837,
                                "i" : 999
                        },
                        ...
                }
        ],
        "ok" : 1
}

    Save this OpTime along with the backup: {t: 1373658837, i: 999}

  4. Start the secondary back up and let it catch up with its replica set.

Import

You need to import this backup into a TokuMX replica set. The latter half of the offline migration process applies here:

  1. Install TokuMX for your distribution on all machines (instructions). If your distribution's package manager automatically starts TokuMX, stop it for now.

  2. Import your backup to just the primary with mongorestore. You will need the dbpath from /etc/tokumx.conf (by default, /var/lib/tokumx).

    Primary server only:

    $ mongorestore --dbpath /var/lib/tokumx /var/lib/mongodb.backup

  3. Add the replSet option to /etc/tokumx.conf on all machines, for example, replSet = rs0.

  4. Start the primary, connect to it, and run rs.initiate() and wait for it to become primary, and then shut it down, to initialize the oplog.

    Primary server only:

    $ mongo
TokuMX mongo shell v1.4.2-mongodb-2.4.10
connecting to: test
> rs.initiate()
{
	"info2" : "no configuration explicitly specified -- making one",
	"me" : "db1.localdomain:27017",
	"info" : "Config now saved locally.  Should come online in about a minute.",
	"ok" : 1
}
>
rs0:PRIMARY>

  5. Copy the dbpath to all the secondaries. The data is already compressed, so compressing with rsync will not be faster, and it will be much faster than a normal initial sync.

  6. Add the fastsync=true option to /etc/tokumx.conf on all secondaries.

  7. Start TokuMX on all servers.

  8. Connect to the primary and rs.add() each of the secondaries. With fastsync they will not need to do a full initial sync.

    Primary server only:

    $ mongo
TokuMX mongo shell v1.4.2-mongodb-2.4.10
connecting to: test
rs0:PRIMARY> rs.add('db2.domain:27017')
{ "ok" : 1 }
rs0:PRIMARY> rs.add('db3.domain:27017')
{ "ok" : 1 }

  9. Remove the fastsync=true option from /etc/tokumx.conf on each of the secondaries. You do not need to restart them now.

Catchup

This phase uses the mongo2toku tool to catch the TokuMX replica set up with the MongoDB replica set by replaying the MongoDB oplog.

Below, suppose the MongoDB replica set is identified by mongodb/v1.domain,v2.domain,v3.domain and the TokuMX replica set is identified by tokumx/t1.domain,t2.domain,t3.domain.

  1. Using the OpTime recorded earlier, start mongo2toku. This will read the MongoDB oplog and replay its operations on the TokuMX replica set, but the TokuMX replica set will not count for write concern on the MongoDB set, and it will not be able to vote in elections on the MongoDB set.

    $ mongo2toku --ts=1373658837:999 \
    --from mongodb/m1.domain,m2.domain,m3.domain \
    --host tokumx/t1.domain,t2.domain,t3.domain

Let mongo2toku run until it is fully caught up to (or only a few seconds behind) the MongoDB replica set. Feel free to stop and start it, taking in to consideration the notes below.

Once mongo2toku is fully synced, it will continue to tail the MongoDB oplog until stopped, so you can leave it running and keep the TokuMX replica set synced until it is appropriate to switch over your application.

Switch

When you are ready to switch your application over to TokuMX, leave mongo2toku running through the entire process.

  1. Pause your application's writes to the MongoDB replica set.

  2. Wait until mongo2toku reports that it is "fully synced" a few times to the log.

  3. Redirect your application to point to the TokuMX replica set.

  4. Shut down mongo2toku.

At this point, you can delete the old MongoDB data and shut down the machines it was running on.

Notes on mongo2toku

It is important that mongo2toku does not try to replay an operation twice. If there is an error, or if mongo2toku is stopped manually (control C works), mongo2toku will save a file in the current directory with the OpTime it had reached while syncing, and will also print that value to the console.

When resuming, it will assume that operation was replayed and will try to replay the following operation. If you do not provide --ts when restarting mongo2toku it will use the value saved in the file, but --ts can override it.

Advanced topics

Testing workloads

The mongo2toku tool is simple: it reads the oplog from one MongoDB server or set of servers and replays those operations on another MongoDB or TokuMX server or set of servers. Therefore, it can be used to test TokuMX's suitability for handling a write workload that MongoDB is running, without making the application rely on TokuMX for queries.

To do this, you would follow the above guide until TokuMX was caught up, then just leave mongo2toku running and monitor system load and metrics, response latencies, and anything else important for your workload. You could also try building additional background indexes on TokuMX.

Since mongo2toku just looks like a normal client application to the TokuMX server, you can run multiple instances copying data to the same TokuMX replica set. Therefore, it is possible to simulate the result of running a MongoDB sharded cluster workload on a single TokuMX replica set. For details on how to migrate a MongoDB sharded cluster to a single TokuMX replica set, see Migrating from MongoDB: Sharded cluster (offline, all data at once).

Rolling migration

The online migration procedure typically requires approximately double the machine capacity since there need to be two replica sets running concurrently: MongoDB and TokuMX.

However, it is possible to do an online migration with little or no extra capacity. This process is more complicated and takes longer, but is suitable for applications with limited resources.

The process is site-specific, but this is the general procedure:

  1. Begin as above, by taking a backup and noting the OpTime of that backup.

  2. Instead of starting the secondary back up, remove it from the MongoDB replica set and delete its dbpath.

    If necessary, add more arbiters to the MongoDB replica set, since it has one fewer members now.

  3. Import the backup into TokuMX on the machine that was just cleaned. Turn it into a replica set (set replSet and run rs.initiate()) and add some arbiters to that replica set.

  4. (Optional) Shut down that TokuMX instance once its oplog is initialized, and copy its dbpath somewhere else, to be used to start other TokuMX secondaries later. Then start it again.

  5. Run mongo2toku as before, from the MongoDB replica set to the TokuMX replica set.

  6. Select another MongoDB secondary, remove it from the replica set, and delete its dbpath.

  7. Install TokuMX to that machine, and either add it to the TokuMX replica set and let it complete an initial sync, or if the original TokuMX dbpath was saved above, copy that to the dbpath and start the server with fastsync to avoid the initial sync (see the offline migration instructions for more details on fastsync).

  8. Continue taking machines out of the MongoDB replica set and replacing them with TokuMX machines until there are enough TokuMX machines to handle the full application workload. During this process, any application queries that can normally be done on secondaries (with slaveOk) can be redirected to the TokuMX replica set if it is suitably well synced.

  9. When the TokuMX replica set is full enough to accept the full application workload, switch the application over to it (remember to wait for mongo2toku to be "fully synced" after pausing the application), and then tear down the remaining MongoDB machines and replace them with TokuMX machines if needed.

Testing the migration process

The dbHash command can be run on any database and will produce a hash value that is dependent on all the documents in that database. After a successful migration, it will produce the same results on MongoDB as on TokuMX, so it can be used to verify that a migration process was implemented successfully. Just remember to make sure MongoDB is not accepting writes while dbHash is running, and make sure that mongo2toku says it is "fully synced" before running dbHash on TokuMX.

The dropDups option

When creating a unique index in MongoDB, it is possible to add the option dropDups. This is an arbitrarily destructive operation, so it was not implemented in TokuMX. Even if it were implemented, there is no way for TokuMX to be sure that it dropped the same documents that MongoDB would have dropped.

Therefore, if an ensureIndex with dropDups: true is encountered by mongo2toku, it will attempt to build the index without that option, but if there actually are duplicates, that index build will fail.

If it fails, the only option is to restart the migration process by taking a new backup. That backup will now be after the index creation happened, so you won’t need to process the same index build operation again.

Quick Links

Clone this wiki locally