MongoDB World 2019: Lessons Learned: Migrating Buffer's Production Database to MongoDB Atlas from MongoDB 2.4

Lessons Learned: 
Migrating Buffer's Production
Database to MongoDB Atlas
from Mongo 2.4 Dan Farrelly

Who am I? What is Buffer?
! Dan Farrelly - CTO
! buffer.com
! Buffer is a suite of products to help you build your brand and connect with
your customers online.
! Buffer Publish - Schedules and sends 800k+ social media posts per day

! MongoDB 2.4.8
! 4+ TB
! 8 Shards across 2 large instances
! Another cloud provider
! Spun up in January 2013
! Sharded in mid 2014
! MongoDB 3.4
! ? TB
! ? Shards
! MongoDB Atlas for ease of
management, future upgrades
! Migration September 2018
Before After

Why we couldn’t use out-of-box tools
! Live migration tools didn’t support migration of sharded databases from
! Database access was locked down from previous cloud provider
! Could not upgrade database in place
! We wanted to change the number of shards (from 8 to 3)
! We wanted to change some of our shard keys
! But we were/are a team of non-MongoDB-experts

Our migration plan
! Migrate large collections (> 10GB or > 5M documents)
! Build indexes (1-2 days)
! Run delta migrations on large collections
! Enter maintenance mode in the app
! Migrate small collections using mongodump + mongorestore
! Run final delta migrations on large collections
! Validate data & indexes
! Cutover application to target database
! Disable maintenance mode

Get your driver versions right
! Check compatibility before you choose your target db version
! Wire Protocol version incompatibility can cause issues
! Be mindful of major driver changes across versions
👍 3.4 2.4
👎 3.6 2.6

Leveraging mongodump
! Use the mongodump and mongorestore versions for your target database -
don’t mix and match!
! You can pipe mongodump in to mongorestore to avoid having to write to
disk in between both steps
! Speed things up w/ --numInsertionWorkersPerCollection
! Skip indexes if you run into issues restoring
mongodump -db “my-app” -c “users” --archive
| mongorestore --archive
--numInsertionWorkersPerCollection=2
--noIndexRestore

Finding bad documents in the source
! Bad documents in 2.4
○ Fields starting with “$”
○ Indexed fields with values greater than 1024 bytes
! Insert all documents into a capped collection in a temp database running
target version

Use Bulk + Delta for largest collections
! Largest collections will be too slow for a mongo dump + restore
! Use bulk and delta migration approach:
○ Bulk insert all data & record timestamp
○ Index a field like updated_at on source collection
○ Find all updated documents and upsert into the target
! Remember to log deleted records in between deltas

Number your scripts
! Feels simple, but makes your migration fool-proof
/scripts
010createIndexes.sh
020staticCollections.sh
100scaleDownWorkers.sh
110enterMaintenanceMode.sh
200tokens.sh
210permissions.sh
220payments.sh
300validation.sh

Double & triple check indexes have been built
! Validate your indexes have been built
! 1 or 2 indexes failing to build could mean hours of headaches for you
! Don’t end up with a collScan on a collection with 1B+ records!
function createAll(){
db.coll.createIndexes([…])
db.other.createIndex({…})
}
createAll()

Train your new database to use indexes (if needed)
! Your new database doesn’t always choose the index you want it to initially
db.<collection>.getPlanCache().clear()
db.<collection>.getPlanCache().clearPlansByQuery(…)
db.runCommand({ planCacheSetFilter: … })

Sharding: Stop the balancer
! Try to ensure your source and target cluster is not trying to balance your
data while you’re migrating
! You don’t want data moving between shards during migration!
! If you’re pre-splitting chunks and stop the balancer you won’t need to
balance during migration on the target anyway 😉
sh.getBalancerState();
sh.isBalancerRunning();
sh.stopBalancer();
sh.setBalancerState(false)

Sharding: Pre-split chunks for faster migration
! Estimate how much of your data should fit within a chunk (64MB)
! Write a script to iterate through your source data and run the split
command on the target collection
for (var i=2012; i<2019; i++) {
for (var j=1; j<=12; j++) {
for (var k=1; k<=31; k++) {
// Our collection had roughly 64MB of data for every 2 hours
// of “profiles” between 2012 and 2018:
for (var hour=2; hour<=24; hour=hour+2) {
db.adminCommand({
split: “buffer.updates”,
middle: { profile_id: ObjectId.fromDate(…), _id: MaxKey } })

Our results
! 4+ TB before to 1.2 TB w/ WiredTiger’s compression
! 8 Shards to 3 shards
! Averaged 40-100 QR/QW during peak time to 0-2
! Maintenance cutover was 4 hours longer than expected due to unforeseen
issues with index building
! Easy upgrade to 3.6 with zero downtime a few months later

And we did it all remotely!
! 11 hours of Zoom video calls during the day of migration
! 1 day of in-person planning with consulting engineer - everything else done
over Slack + Zoom
! If we can perform such a gnarly migration without being MongoDB
experts, so can you!

Questions?
dan@buffer.com
@djfarrelly

MongoDB World 2019: Lessons Learned: Migrating Buffer's Production Database to MongoDB Atlas from MongoDB 2.4

Recommended

More Related Content

What's hot (20)

Similar to MongoDB World 2019: Lessons Learned: Migrating Buffer's Production Database to MongoDB Atlas from MongoDB 2.4 (20)

More from MongoDB (20)

Recently uploaded (20)

MongoDB World 2019: Lessons Learned: Migrating Buffer's Production Database to MongoDB Atlas from MongoDB 2.4