Postgres in Production - Best Practices 2014
20 likes29,458 views
This presentation explores a broad cross-section of enterprise Postgres deployments to identify key usage patterns and reveals important aspects of performance, scalability, and availability including: * Challenges organizations encounter most frequently during the stages of database development, deployment and maintenance * Tuning parameters used most frequently to improve performance of production databases * Frequently problematic database maintenance processes and configuration parameters * Most commonly-used database back-up and recovery strategies
More Related Content
What's hot (20)
Similar to Postgres in Production - Best Practices 2014 (20)
More from EDB (20)
Recently uploaded (20)
Postgres in Production - Best Practices 2014
- 1. © 2014 EDB All rights reserved. 1 Postgres in Production – Enterprise Best Practices
- 2. © 2014 EDB All rights reserved. 2 • Explore key Postgres usage patterns, to expose − Challenges users face in adoption, development, deployment and maintenance − Frequently used tuning parameters − Common database maintenance practices − Database back-up and recovery strategies • Provide empirical data to help you anticipate needs during your database lifecycles • Share ideas and resources to help you address problems before they occur Objectives
- 3. © 2014 EDB All rights reserved. 3 • EnterpriseDB Overview (quick) • Postgres Support Experiences Explore what kinds of issues your peers are experiencing with Postgres, and when they’re encountering those issues • Database Health Checks Share what we’ve learned in the field about how Postgres databases are deployed, tuned and supported • Summary and Useful Resources • Q&A Agenda We will take two brief polls during the webinar to gather some group info about your Postgres usage and areas of need
- 4. © 2014 EnterpriseDB Corporation. All rights reserved. 4 Brief EDB Overview
- 5. © 2014 EnterpriseDB Corporation. All rights reserved. 5 Postgres Plus Advanced Server Postgres Plus Cloud Database High Availability PerformanceManagement REMOTE DBA 24x7 SUPPORT PROFESSIONAL SERVICES TRAINING EDB Serves All Your Postgres Needs PostgreSQL Security
- 6. © 2014 EnterpriseDB Corporation. All rights reserved. 6 POSTGRES innovation ENTERPRISE reliability 24/7 support Services & training Enterprise-class features & tools Indemnification Product road-map Control Thousands of developers Fast development cycles Low cost No vendor lock-in Advanced features Enabling commercial adoption of Postgres
- 7. © 2014 EnterpriseDB Corporation. All rights reserved. 7 Postgres Support Experiences
- 8. © 2014 EnterpriseDB Corporation. All rights reserved. 8 • Adoption of Postgres is growing fast − More and more Postgres users are not not ‘dyed in the wool’ Postgres users – they come from Oracle, MS SQL, DB2, etc. • Postgres is moving into complex, mission-critical applications − Question patterns are changing from simple to complex − Expertise level is changing – when the customer calls, it is with a (more) difficult problem or urgent issue • How do we make Postgres users even more successful? − Targeted white papers − Targeted training for support team − Pro-active training for users as part of our customer enablement process − Balanced product strategy of new features, new capabilities and usability Why We Analyze Support Tickets
- 9. © 2014 EnterpriseDB Corporation. All rights reserved. 9 Ticket Volumes & Multi Year Growth • Q1 2014: ~ 2500 total customers • Analysis in this presentation is based on a representative sample of tickets raised over a 12 months period
- 10. © 2014 EnterpriseDB Corporation. All rights reserved. 10 Observations • Postgres Plus Advanced Server: Strong emphasis on proprietary or commercially supported OSS • PostgreSQL: Significantly less emphasis on proprietary or commercially supported OSS – even though the customer has commercially supported Postgres Distribution Tickets and Platforms
- 11. © 2014 EnterpriseDB Corporation. All rights reserved. 11 When do customers call?
- 12. © 2014 EnterpriseDB Corporation. All rights reserved. 12 What Do Customers Call About? 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% POC Dev Deployment Maintenance Tuning Product Awareness How to Enhancements Corruption Bug # Tickets Phase Category POC Dev Deployment Maintenance Grand Total Bug 8% 22% 9% 7% 10% Corruption 0% 0% 0% 6% 2% Enhancements 3% 4% 1% 0% 1% How to 41% 33% 41% 29% 35% Product Awareness 46% 39% 44% 39% 41% Tuning 3% 2% 5% 19% 10% Grand Total 100% 100% 100% 100% 100%
- 13. © 2014 EnterpriseDB Corporation. All rights reserved. 13 Where Do Users Need Help # Tickets Phase Category POC Dev Deployment Maintenance Grand Total Connectors 0% 2% 1% 0% 3% Bug 0% 0% 0% 0% 1% How to 0% 1% 0% 0% 1% Product Awareness 0% 1% 0% 0% 1% Database 10% 9% 17% 37% 73% Bug 1% 2% 1% 2% 6% Corruption 0% 0% 0% 2% 2% Enhancements 0% 0% 0% 0% 1% How to 4% 3% 6% 10% 23% Product Awareness 4% 3% 8% 15% 31% Tuning 0% 0% 1% 7% 9% Replication 1% 4% 4% 2% 12% Bug 0% 1% 0% 0% 2% How to 0% 0% 3% 1% 4% Product Awareness 1% 2% 1% 0% 4% Tuning 0% 0% 0% 1% 1% Utilities 1% 3% 6% 3% 12% Bug 0% 1% 1% 0% 2% Enhancements 0% 0% 0% 0% 0% How to 1% 1% 3% 1% 6% Product Awareness 0% 1% 3% 1% 5% Grand Total 12% 17% 28% 42% 100%
- 14. © 2014 EnterpriseDB Corporation. All rights reserved. 14 Deployment/Database/Product Awareness & How To − How to set up connection pooling? − How to set up Hot Standby and Streaming Replica? − How to add table spaces? − How to partition? − How to upgrade from major version? Maintenance/Database/Product Awareness & How To − How to find bloat in tables & indexes? − How to enable auditing selectively without performance impact? − What are the effects of changes to the values of postgresql.conf file parameters wal_writer_delay, bgwriter_delay? − What is the correct configuration of hba_conf? Select Top questions (not in rank order)
- 15. © 2014 EnterpriseDB Corporation. All rights reserved. 15 • The majority of questions arise after PoC and Development • Deployment and Maintenance are the more challenging phases • Plan early to understand necessary practices − Backup, recovery, high availability strategies − Configuration parameter tuning to support growth − Vacuum and bloat − Connection pooling Best Practices and Lessons from Support
- 16. © 2014 EnterpriseDB Corporation. All rights reserved. 16 Lessons Learned from the AHC
- 17. © 2014 EnterpriseDB Corporation. All rights reserved. 17 • Systematic top-to-bottom review of the customer’s Postgres production installation • Identifies issues and improvement opportunities in: − Performance − Scalability and support for business and data growth − Availability – Backup strategies, recovery, replication and failover − Maintenance and tuning − Upgrades • The following analysis is based on a set of 20 AHCs executed in 2013 Architectural Health Check
- 18. © 2014 EnterpriseDB Corporation. All rights reserved. 18 0%# 10%# 20%# 30%# 40%# 50%# 60%# 70%# 80%# 90%# 100%# effec0ve_cache_size# shared_buffers# bgw riter_lru_m axpages# w al_buffers# m aintenance_w ork_m em # bgw riter_lru_m ul0plier# w ork_m em # random _page_cost# checkpoint_segm ents# cpu_tuple_cost# checkpoint_com ple0on_target# checkpoint_0m eout# log_line_prefix# log_lock_w aits# log_m in_dura0on_statem ent# autovacuum _vacuum _scale_factor# checkpoint_w arning# autovacuum _vacuum _cost_delay# autovacuum _vacuum _threshold# autovacuum _nap0m e# log_autovacuum _m in_dura0on# Top$Database$Parameter$Adjustments$3$Ordered$by$Frequency$ Red bars: experts expected top parameters, including vacuum_cost_limit, log_line_prefix, log_min_duration_statement, log_checkpoints Top Database Parameter Adjustments – Ordered by Frequency
- 19. © 2014 EnterpriseDB Corporation. All rights reserved. 19 Kernel Parameters 0 2 4 6 8 10 12 14 16
- 20. © 2014 EnterpriseDB Corporation. All rights reserved. 20 Backup and Standby hot 47% warm 23% None 18% cold 12% Standby Approach Physical Only 36% Logical Only 29% Physical and Logical 14% FS Snapshot 7% None 14% Backup Approach 43% of customers could not execute PITR (prior to AHC) – some had NO backup in place 30% of customers would have been slow or unable to recover from failure
- 21. © 2014 EnterpriseDB Corporation. All rights reserved. 21 Maintenance Tasks Unused Indexes 39% Bloated Indexes 30% Bloated Tables 13% Missing Indexes 9% Missing PK 9% Bloat management (43%) is a key production problem specific to Postgres
- 22. © 2014 EnterpriseDB Corporation. All rights reserved. 22 • Majority of Postgres production deployments are on Linux based OS • Storage system is a key deployment consideration – DAS/SAN/NFS • OS System Adjustments (dirty_background_ratio & dirty_ratio) are also a major adjustment (70% & 45%) Observations from the AHCs
- 23. © 2014 EnterpriseDB Corporation. All rights reserved. 23 Stage: post-production Primary Issue: • Outage due to replication failure - insufficient log info to determine true root cause Potential Pain Points • Concerns have reached limits of scale and meeting anticipated growth demands Recommendations • Backup and recovery strategies • Implement continuous archive recovery in addition to streaming replication • Revised backup strategy with PITR • Tune auto-vacuum settings and manually vacuum large tables • Indexes – create concurrent indexes, REINDEX to reduce bloat • Parameter tuning – various for kernel, memory settings and WALs AHC Case Study #1 Use Case: on-line advertising service with hosted chat
- 24. © 2014 EnterpriseDB Corporation. All rights reserved. 24 Stage: pre-deployment Primary Issues: • Performance in returning results and timeouts • No backup strategy in place Potential Pain Points • Expected data volume growth and increased analysis – scalability concerns Recommendations • Kernel param updates (8)- kernel.shmmax, kernel.shmall, vm.dirty_ratio, vm.dirty_background_ratio,…! • Database param changes (11) - effective_cache_size, shared_buffers, bgwriter_lru_maxpages, wal_buffers, work_mem, … • Re-architect specific heavily- used database function • Detailed backup and availability strategies AHC Case Study #2 Use Case: data warehouse reporting at customer sites
- 25. © 2014 EnterpriseDB Corporation. All rights reserved. 25 Stage: post-production Primary Issues: • System performance • Running out of connections • Slow queries/ locked tables • Advice on new SW release Potential Pain Points • Concern for increased workloads and scalability Recommendations • max_connections was actually too high, work_mem too low • Increase checkpoint parameters − checkpoint_segments, checkpoint_timeout, checkpoint_completion_ target, checkpoint_warning • Tune storage-related params (SAN) - DB, memory, file system work_mem, random_page_cost • Upgrade for performance and security improvements • As workload increases, increase RAM and re-tune related memory config. parameters AHC Case Study #3 Use Case: claims mgmt. system and reporting application
- 26. © 2014 EnterpriseDB Corporation. All rights reserved. 26 Summary and Useful Resources
- 27. © 2014 EnterpriseDB Corporation. All rights reserved. 27 • Postgres users consistently experience different kinds of challenges throughout the database lifecycle − Postgres product knowledge is a critical need at all lifecycle stages • Many DBAs would benefit from better knowledge of tuning Postgres for their storage infrastructures • Tuning databases early in the lifecycle can save a lot of time, hassle and user friction after they’re deployed • Many organizations lack appropriate backup and recovery strategies Summary
- 28. © 2014 EnterpriseDB Corporation. All rights reserved. 28 • Why do you need backup and recovery? Backup and Recovery Strategies protect you in case of: − Catastrophic Device Failure − Site Failure − Maintenance − Operator Error − Compliance − Data Corruption • Key takeaways: − Logical backup provides granularity in objects (tables, table spaces, databases); Physical backups provide granularity in time for PITR. − Understand requirements before diving into technology solution Backup and Recovery Strategies
- 29. © 2014 EnterpriseDB Corporation. All rights reserved. 29 • Postgres provides − Logical backup (pg_dump) − Binary/physical backup (Filesystem Snapshots & pg_basebackup) − WAL (write ahead logs) − Streaming replication − Delayed replication • Advanced solutions, such as RHCS, Veritas Clustering, EFM and EDB BART - (now in Beta) augment the basic capabilities Backup and Recovery Strategies
- 30. © 2014 EnterpriseDB Corporation. All rights reserved. 30 • Performance, Scalability, Availability • Three-day in-depth Postgres database review • Analysis of platform, database architecture and application usage • Provides tuning and management guidance based on EDB’s expert best practices Get an Architectural Health Check when you are − Not 100% sure about your backup/recovery strategy − Unsure your database is optimally tuned for your apps − Moving Postgres databases to the Cloud − Upgrading your servers to multi-core, high memory devices − Experiencing high growth in application usage − Experiencing deterioration of read and/or write throughput Architectural Health Check http://enterprisedb.com/services/packaged-services/health-check
- 31. © 2014 EnterpriseDB Corporation. All rights reserved. 31 For more information about an Architectural Health Check, please visit www.enterprisedb.com/ahc or email us - [email protected]