MongoDB World 2019: Lessons Learned: Migrating Buffer's Production Database to MongoDB Atlas from MongoDB 2.4

MongoDB World 2019: Lessons Learned: Migrating Buffer's Production Database to MongoDB Atlas from MongoDB 2.4

• 1.
  Lessons Learned:
 Migrating Buffer'sProduction Database to MongoDB Atlas from Mongo 2.4 Dan Farrelly
• 2.
  Who am I?What is Buffer? ! Dan Farrelly - CTO ! buffer.com ! Buffer is a suite of products to help you build your brand and connect with your customers online. ! Buffer Publish - Schedules and sends 800k+ social media posts per day
• 3.
  ! MongoDB 2.4.8 !4+ TB ! 8 Shards across 2 large instances ! Another cloud provider ! Spun up in January 2013 ! Sharded in mid 2014 ! MongoDB 3.4 ! ? TB ! ? Shards ! MongoDB Atlas for ease of management, future upgrades ! Migration September 2018 Before After
• 4.
  Why we couldn’tuse out-of-box tools ! Live migration tools didn’t support migration of sharded databases from ! Database access was locked down from previous cloud provider ! Could not upgrade database in place ! We wanted to change the number of shards (from 8 to 3) ! We wanted to change some of our shard keys ! But we were/are a team of non-MongoDB-experts
• 5.
  Our migration plan !Migrate large collections (> 10GB or > 5M documents) ! Build indexes (1-2 days) ! Run delta migrations on large collections ! Enter maintenance mode in the app ! Migrate small collections using mongodump + mongorestore ! Run final delta migrations on large collections ! Validate data & indexes ! Cutover application to target database ! Disable maintenance mode
• 6.
  The Basics
• 7.
  Get your driverversions right ! Check compatibility before you choose your target db version ! Wire Protocol version incompatibility can cause issues ! Be mindful of major driver changes across versions 👍 3.4 2.4 👎 3.6 2.6
• 8.
  Leveraging mongodump ! Usethe mongodump and mongorestore versions for your target database - don’t mix and match! ! You can pipe mongodump in to mongorestore to avoid having to write to disk in between both steps ! Speed things up w/ --numInsertionWorkersPerCollection ! Skip indexes if you run into issues restoring mongodump -db “my-app” -c “users” --archive | mongorestore --archive --numInsertionWorkersPerCollection=2 --noIndexRestore
• 9.
  Finding bad documentsin the source ! Bad documents in 2.4 ○ Fields starting with “$” ○ Indexed fields with values greater than 1024 bytes ! Insert all documents into a capped collection in a temp database running target version
• 10.
  Use Bulk +Delta for largest collections ! Largest collections will be too slow for a mongo dump + restore ! Use bulk and delta migration approach: ○ Bulk insert all data & record timestamp ○ Index a field like updated_at on source collection ○ Find all updated documents and upsert into the target ! Remember to log deleted records in between deltas
• 11.
  Number your scripts !Feels simple, but makes your migration fool-proof /scripts 010createIndexes.sh 020staticCollections.sh 100scaleDownWorkers.sh 110enterMaintenanceMode.sh 200tokens.sh 210permissions.sh 220payments.sh 300validation.sh
• 12.
  Double & triplecheck indexes have been built ! Validate your indexes have been built ! 1 or 2 indexes failing to build could mean hours of headaches for you ! Don’t end up with a collScan on a collection with 1B+ records! function createAll(){ db.coll.createIndexes([…]) db.other.createIndex({…}) } createAll()
• 13.
  Train your newdatabase to use indexes (if needed) ! Your new database doesn’t always choose the index you want it to initially db.<collection>.getPlanCache().clear() db.<collection>.getPlanCache().clearPlansByQuery(…) db.runCommand({ planCacheSetFilter: … })
• 14.
  The Advanced
• 15.
  Sharding: Stop thebalancer ! Try to ensure your source and target cluster is not trying to balance your data while you’re migrating ! You don’t want data moving between shards during migration! ! If you’re pre-splitting chunks and stop the balancer you won’t need to balance during migration on the target anyway 😉 sh.getBalancerState(); sh.isBalancerRunning(); sh.stopBalancer(); sh.setBalancerState(false)
• 16.
  Sharding: Pre-split chunksfor faster migration ! Estimate how much of your data should fit within a chunk (64MB) ! Write a script to iterate through your source data and run the split command on the target collection for (var i=2012; i<2019; i++) { for (var j=1; j<=12; j++) { for (var k=1; k<=31; k++) { // Our collection had roughly 64MB of data for every 2 hours // of “profiles” between 2012 and 2018: for (var hour=2; hour<=24; hour=hour+2) { db.adminCommand({ split: “buffer.updates”, middle: { profile_id: ObjectId.fromDate(…), _id: MaxKey } })
• 17.
  Our results ! 4+TB before to 1.2 TB w/ WiredTiger’s compression ! 8 Shards to 3 shards ! Averaged 40-100 QR/QW during peak time to 0-2 ! Maintenance cutover was 4 hours longer than expected due to unforeseen issues with index building ! Easy upgrade to 3.6 with zero downtime a few months later
• 18.
  And we didit all remotely! ! 11 hours of Zoom video calls during the day of migration ! 1 day of in-person planning with consulting engineer - everything else done over Slack + Zoom ! If we can perform such a gnarly migration without being MongoDB experts, so can you!
• 19.
  Questions? dan@buffer.com @djfarrelly