Download as PDF, PPTX
Uploaded byValeri Karpov
MongoDB Miami Meetup 1/26/15: Introduction to WiredTiger
This document provides an overview of WiredTiger and the MongoDB storage engine API. It discusses how WiredTiger differs from the mmapv1 storage engine in its use of document-level locking, compression, and consistency without journaling. It also covers WiredTiger internals like checkpoints, configuration options, and basic performance comparisons showing WiredTiger can provide higher throughput than mmapv1 for write-heavy workloads.
More Related Content
Similar to MongoDB Miami Meetup 1/26/15: Introduction to WiredTiger
MongoDB Miami Meetup 1/26/15: Introduction to WiredTiger
- 1. Introduction to WiredTigerand the Storage Engine API Valeri Karpov NodeJS Engineer, MongoDB www.thecodebarbarian.com www.slideshare.net/vkarpov15 github.com/vkarpov15 @code_barbarian
- 2. * A Bit AboutMe •CI/NodeJS Engineer at MongoDB •Maintainer of mongoose ODM •Recently working on rewriting mongodump, etc.
- 3. * Talk Overview •What isthe Storage Engine API? •What is WT and why you should care •Basic WT internals and gotchas •MMS Automation and WT •Some very basic performance numbers
- 4. * Introducing Storage Engines •HowMongoDB persists data •<= MongoDB 2.6: “mmapv1” storage engine •MongoDB 3.0 has a few options • mmapv1 • in_memory • wiredTiger • devnull (now /dev/null does support sharding!) •Internal hack: Twitter storage engine
- 5. * Why Storage EngineAPI? •Different performance characteristics •mmapv1 doesn’t handle certain workloads well •Consistent API on top of storage layer • Can mix storage engines in a replset or sharded cluster!
- 6. * Storage Engines Visually •mmapv1persists data to dbpath
- 7. * Storage Engines Visually •WTpersists data differently
- 8. * What is WiredTiger? •Storageengine company founded by BerkleyDB alums •Recently acquired by MongoDB •Available as a storage engine option in MongoDB 3.0
- 9. * Why is WiredTigerAwesome? •Document-level locking •Compression on disk •Consistency without journaling •Better performance on certain workloads
- 10. * Document-level Locking •The often-criticizedglobal write lock was removed in 2.2 •Database-level locking •3.0 with mmapv1 has collection-level locking •3.0 with WT only locks at document layer •Writes no longer block all other writes •Better CPU usage: more cores ~= more writes
- 11. * Compression •WT uses snappycompression by default •Data is compressed on disk •2 supported compression algorithms: • snappy: default. Good compression, relatively low overhead • zlib: Better compression, but at cost of more overhead
- 12. * Consistency without Journaling •mmapv1uses write-ahead log to guarantee consistency as well as durability •WT doesn’t have this problem: no in-place updates •Potentially good for insert-heavy workloads •Rely on replication for durability •More on this in the next section
- 13. * WiredTiger Internals: RunningIt •mongod now has a --storageEngine option
- 14. * WiredTiger and dbpath •Don’trun WT when you have mmapv1 files in your dbpath (and vice versa)
- 15. * Upgrading to WT •Can’tcopy database files •Can’t just restart with same dbpath •Other methods for upgrading still work: • Initial sync from replica set • mongodump/mongorestore •Can still do rolling upgrade of replica set to WT: • Shut down secondary, delete dbpath, bring it back up with -- storageEngine wiredTiger
- 16. * Other Configuration Options •Compression:--wiredTigerCollectionBlockCompressor •
- 17. * Other Configuration Options •directoryperdb:doesn’t exist in WT •Databases are a higher level abstraction with WT •Following options also have no WT equivalent • noprealloc • syncdelay • smallfiles • journalCommitInterval
- 18. * Configuration with YAMLFiles •MongoDB 2.6 introduced YAML config files •The storage.wiredTiger field lets you tweak WT options
- 19. * WiredTiger Journaling •Journaling inWT is a little different •Write-ahead log committed to disk at checkpoints •By default checkpoint every 60 seconds or 2GB written •Data files always consistent - no journaling means you lose data since last checkpoint •No journal commit interval: writes are written to journal as they come in
- 20. * Gotcha: system.indexes, system. namespacesdeprecated •Special collections in mmapv1 •Explicit commands: db.getIndexes(), db. getCollectionNames()
- 21. * Gotcha: No 32-bitSupport •WT storage engine will not work on 32-bit platforms at all
- 22. * Using MMS Automationwith WT •MMS automation allows you to manage and deploy MongoDB installations •Demo of upgrading a standalone to WT
- 23. * Some Basic PerformanceNumbers •My desired use case: MongoDB for analytics data •Write-heavy workloads aren’t mmapv1’s strong suit •Don’t care about durability as much, but do care about high throughput •Compression is a plus •How does WT w/o journaling do on insert-only? •Simple N=1 experiment
- 24. * Some Basic PerformanceNumbers •mongo shell’s benchRun function
- 25. * Some Basic PerformanceNumbers •mmapv1 WT with --nojournal
- 26. * Some Basic PerformanceNumbers •How does the compression work in this example? •After bench run, WT’s /data/db sums to ~23mb •With --noprealloc, --nopreallocj, --smallfiles, mmapv1 has ~100mb •Not a really fair comparison since data is very small
- 27. * Thanks for Listening! •Slideson: • Twitter: @code_barbarian • Slideshare: slideshare.net/vkarpov15