More Related Content
What's hot (18)
Viewers also liked (7)
Similar to Operationalizing MongoDB at AOL (20)
Recently uploaded (20)
Operationalizing MongoDB at AOL
- 1. āOperationalizingā MongoDB@AOL Michael DelNegro Principal Database Administrator AOL
- 2. About Me ā¢ DBA at AOL (Dulles) for 6+ years ā¢ Original DBMS Background is in Sybase ā¢ Now MySQL, MongoDB, PostgreSQL and NoSQL (Neo4J, Redis, etc) ā¢ Check out www.meetup.com/NOVA- MySQL
- 3. The First Mongo at AOL ā¢ A MapQuest Developer in the Summer of 2010 Replaced a MySQL Datastore with Mongo ā¢ That Developer left in early 2011 :-/ ā¢ Was Handed Over to Operations and Yours Truly
- 4. Never Forget Your First ā¢ Not Small: 40+ Hosts ā¢ Upgrade Challenges ā¢ Connection Floods ā¢ Support to the Rescue ā¢ Magnified the Reasons to Operationalize
- 6. Operation āOperationalizeā ā¢ Take Advantage of 10Gen Support ā¢ Establish Standards ā¢ Monitoring/Alerting ā¢ Backups ā¢ Information Sharing ā¢ Challenges
- 7. 10Gen Support ā¢ JIRA Documents Cases Enabling Information Sharing and Referencing ā¢ Serves Us Well in Both āEmergenciesā and āHey Quick Questionā ā¢ Training Classes: Administration & Development
- 8. Standard Host Setup ā¢ CentOS 6 ā¢ XFS/EXT4 ā¢ Deadline I/O Scheduler ā¢ dirty_ratio = 10 (down from 40) ā¢ dirty_background_ratio = 5 (down from 10) ā¢ noatime
- 9. Standard Build Scripts ā¢ Use 64bit Linux Binaries Stored in AOL Repository ā¢ mongodb_install.sh ā Parameters Include: ā¢ MongoDB Version ā¢ Replica Set Name ā¢ Sharded? (Y/N) ā¢ Port (27017 is standard) ā¢ Type (DB, Arbiter, Config, Mongos)
- 10. Standard Directory/Layouts ā¢ Based on our standard MySQL layout ā¢ Binaries in /opt/local/pkg ā¢ /db/mongodb/<port> base directory ā Ex. /db/mongodb/27017 ā Can Override ā¢ /db/mongodb/27017-repset1 ā¢ Under base directory ā data ā etc (config files) ā logs ā scripts (ex stop/start)
- 11. ORB ā¢ AOL Technologiesā Configuration Management Database (CMDB) ā¢ Integrated with many authoritative data repositories ā¢ Unique namespace for many operations data points ā¢ Data model for operations management ā¢ Projects, Assets, People, Applications, Network Data, HCM ā¢ SQL Interface
- 12. Monitoring/Alerting ā¢ Argus ā¢ Nagios ā¢ Component Test Tool ā¢ MMS (MongoDB Monitoring Service)
- 13. Argus ā¢ Metric and Event ā Collector ā Thresholds ā Management ā Data Viewing
- 14. Argus ā¢ Capture serverStatus Metrics (once a minute) ā¢ Capture Replication Lag ā¢ Great for Trending ā¢ Great for Capacity Planning ā¢ Great for Troubleshooting ā¢ We Also Use for Host Metrics (CPU, I/O, etc) ā¢ Administrated By a Small and very Busy Group ā Requested Additions/Changes Can Be Slow
- 15. Nagios ā¢ Great For Fault-detection/Alerting ā¢ Great For Show Me What Is Currently Broken ā¢ Great For Service Availability Metrics ā¢ Flexible ā¢ Reduces Pressure On NOC ā¢ Integrates With Netcool, Ignore Tool ā¢ We Write Our Own Plug-Ins
- 16. Component Test Tool ā¢ Runs Parallel HTTP GETs Across a Group of Hosts ā¢ Matches Regular Expressions On the Page Content (or HTTP Headers) ā¢ Displays Number (or Content) of Matches in a HTML Table
- 17. MMS
- 18. Backups ā¢ Scripts Delivered With Install ā¢ Choice Between LVM (default) and Standard (mongodump) ā¢ Ability to Backup Sharded Setup
- 19. Mongo @ AOL Currently
- 20. Mongo @ AOL Currently ā¢ 30+ Active MongoDB Projects ā¢ 500+ Servers in Production ā¢ Developed in: ā Java ā PHP ā Ruby ā C# ā Erlang ā Scala ā Perl ā C++
- 21. Information Sharing ā¢ Internal Wikis/Websites ā¢ UnUniversity (Organized Brown Bags) ā¢ Architecture Reviews ā¢ 10Gen JIRA Access
- 22. Challenges ā¢ Developers Love It ā¢ BUT Requires More Upfront Planning Than They May Realize ā¢ Should NOT Be Our Default Datastore ā¢ Better Developer OnBoarding ā¢ Aggressive Upgrading Is Required ā¢ Coordination of Driver Upgrades
- 23. The Future ā¢ āMongoDB As a Serviceā ā¢ Puppet or Chef ā¢ Internal Administration Console ā Already has Visual Topologies ā¢ Internal MongoDB Operations Class ā First One is in July
- 24. Resources ā¢ Recommended Presentation: Todd Dampierās Rock-Solid Mongo Ops (mongolab) ā¢ Check out Jon Reedās MongoDB for Online Advertising at AOL (MongoDB UK 2012) ā¢ www.meetup.com/Washington-DC- MongoDB-Users-Group/
- 25. Thank You! ā¢ www.slideshare.net/radiocats ā¢ @radiocats ā¢ www.linkedin.com/in/mdelnegro ā¢ Thank you, SeniorGif