TechEvent PostgreSQL Best Practices

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
PostgreSQL Best Practices
Overview from the initial setup to an OLTP performance benchmark
against Oracle.
Emiliano Fusaglia Principal Consultant
Jacques Kostic Principal Consultant

2018 © Trivadis
Exadata X7-2 POC with OVM
2
TechEvent September 2018
Specialties:
• Database Cloud computing (DBaaS)
• Oracle RAC
• Grid Infrastructure (CRS, ASM)
• Data Guard
• Instance and SQL Performance & Tuning
• Linux & Virtualization
Certifications:
• Oracle Certified Professional 9i, 10g, 11g & 12c
• Oracle Exadata Administrator X3 –X4 Certified Expert
Teaching Courses at Trivadis:
• Oracle 11g & 12c Grid Infrastructure & RAC
• Oracle 11g & 12c Data Guard
• Oracle Exadata
• Oracle 12c New Features
About me…
@EFusaglia

PostgreSQL Best Practices3 01/10/2018
Experience:
• Initially C/C++ developer
• In touch with Oracle since 1990 from version 4 on SCO Unix!
• High Availability and Backup & Recovery Architect
• SQL and Instance Performance & Tuning
• License Audit and Consolidation
Certifications:
• Oracle Certified Master 11g & 12c
• Oracle 11g Performance Tuning Certified Expert
• Oracle RAC 11g and Grid Infrastructure Administration
• Oracle Exadata Administrator Certified Expert
• Oracle Certified SQL Expert 11g
Teaching Courses at Trivadis:
• Oracle 11g & 12c Grid Infrastructure & RAC
• Oracle 11g & 12c Data Guard
• Oracle 11g & 12c Performance & Tuning
• Oracle 11g & 12c Administration
• SQL & PL-SQL
• OEM – 12 & 13
About me…
@JKOFR

Agenda
PostgreSQL Best Practices10/1/2018
1. PostgreSQL
Introduction & Architecture
OS Requirements
Installation Options
Securing PostgreSQL ClusterDB
Main parameters to configure
Backup and Recovery
2. OLTP performance benchmark PostgreSQL vs Oracle
Configuration
Results
3. Conclusion
Takeaway
4

PostgreSQL Introduction &
Architecture
5

Introduction to PostgreSQL
PostgreSQL is an opensource and independent Object-RDBMS developed and
maintained by the PostgreSQL Global Development Group.
The first version was released in 1996 as INGRES development, and it included support
for Object orientation and SQL.
Main Characteristics:
ACID (Atomicity, Consistency, Isolation, Durability)
Multiversion concurrency control (MVCC)
Foreign keys, Indexes, Views, Trigger, Functions, Procedural Languages (PL), etc..
Streaming Replication (as of 9.0)
Hot Standby (as of 9.0)

PostgreSQL Architecture
pg_crl start
Source PostgreSQL documentation

Database Cluster
Source PostgreSQL documentation
A cluster is an instance of postgreSQL containing one or many databases
– Conceptually similar to MySQL, SQL Server and Oracle Pluggable Databases
Server
Cluster pgclu01 (port 5438) Cluster pgclu02 (port 5439)
postgres template0 template1 postgres template0 template1
ecom01 erp01 sales01 dwh01 hr01 supp01

Database Cluster
postgres
template0
template1
System or Master database, it contains system tables, views, procedures,
metadata, user and role definitions.
Template0 it is a read-only empty database used as seed database.
Template1 it is a read-write database, which allows customizations before
to be used as default seed database.
App
Application Database it contains application objects like tables, indexes,
views, procedures, constraints etc..

Before Image and Vacuum Process
PostgreSQL has no rollback segments, and it guarantees read consistency in the
following way:
Writing new image in a new location
Flagging the initial image as OLD and keeping intact the data.
Adding a pointer to the OLD image pointing the new one.

Page x
Case 1 The new image remains on the
same page.
1,’blue’
2,’green’
0,’red’
Case 2 The new image migrates
on a new page.
1,’white’ NEW
OLD
UPDATE app_tab SET col2=‘white’ WHERE col0=1;
Page x Page y
1,’bb’,test4
OLD
NEW
1,’blue’
2,’green’
0,’red’
1,’white’
UPDATE app_tab SET col2=‘white’ WHERE col0=1;

VACUUM Process
Reclaims space occupied by old tuple images
Updates data statistics used by the query planner
Updates the visibility map
Resets the transaction ID of old blocks to prevent wraparound
The standard VACUUM is executed regularly by default
Manual VACUUMing is possible

Hands on set up
13

OS Optimization 1/2
PostgreSQL14 01/10/2018
Kernel optimization
– /etc/sysctl.d/90-postgres-sysctl.conf
fs.file-max = 6815744
fs.aio-max-nr = 1048576
kernel.sem = 250 32768 1000 128
kernel.shmall = 1073741824
kernel.shmmni = 4096
kernel.shmmax = 4398046511104
#Huge Pages 80GB
vm.nr_hugepages= 40960
vm.min_free_kbytes=524288

OS Optimization 2/2
PostgreSQL15 01/10/2018
– /etc/security/limits.d/postgres-limits.conf
postgres soft nofile 16384
postgres hard nofile 16384
postgres soft memlock 83886080
postgres hard memlock 83886080
Storage
– Binaries, ClusterDB, External Tablespaces and WAL files on T1 Storage
– Backups on T2 Storage

Installation Options
List of Supported Platforms: Linux (all recent distributions), Windows (Win2000 SP4
and later), FreeBSD, OpenBSD, NetBSD, Mac OS X, AIX, HP/UX, IRIX, Solaris, Tru64
Unix, and UnixWare.
P.S.: this presentation focuses on Linux 64-bit
PostgreSQL can be installed using one of the following method :
Source Code
Binary Package
Binaries Archive without installer (Advanced users) TVD Recommended Option

Securing PostgreSQL ClusterDB
After the installation one of the first tasks to perform is securing the local and remote
database connections, defining the open ports and the authentication methods.
Those settings can be defined in two different configuration files:
postgresql.conf section CONNECTIONS AND AUTHENTICATION
pg_hba.conf

Securing PostgreSQL ClusterDB - postgresql.conf 1/2
#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------
# - Connection Settings -
listen_addresses = '192.168.1.129,localhost' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost'; use '*' for all
# (change requires restart)
port = 5544 # (change requires restart)
max_connections = 100 # (change requires restart)
#superuser_reserved_connections = 3 # (change requires restart)
#unix_socket_directories = '/tmp' # comma-separated list of directories
#unix_socket_group = '' # (change requires restart)
unix_socket_permissions = 0770 # begin with 0 to use octal notation
#bonjour = off # advertise server via Bonjour
#bonjour_name = '' # defaults to the computer name
# - Security and Authentication -
#authentication_timeout = 1min # 1s-600s
#ssl = off
#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
#ssl_prefer_server_ciphers = on
#ssl_ecdh_curve = 'prime256v1'
#ssl_dh_params_file = ''
#ssl_cert_file = 'server.crt'
...

Securing PostgreSQL ClusterDB - postgresql.conf 2/2
# - Security and Authentication -
#authentication_timeout = 1min # 1s-600s
#ssl = off
#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
#ssl_prefer_server_ciphers = on
#ssl_ecdh_curve = 'prime256v1'
#ssl_dh_params_file = ''
#ssl_cert_file = 'server.crt'
#ssl_key_file = 'server.key'
#ssl_ca_file = ''
#ssl_crl_file = ‚‘
password_encryption = scram-sha-256 # md5 or scram-sha-256
#db_user_namespace = off
#row_security = on
# GSSAPI using Kerberos
#krb_server_keyfile = ''
#krb_caseins_users = off
# - TCP Keepalives -
# see "man 7 tcp" for details
#tcp_keepalives_idle = 0 # TCP_KEEPIDLE, in seconds;
# 0 selects the system default
#tcp_keepalives_interval = 0 # TCP_KEEPINTVL, in seconds;
#tcp_keepalives_count = 0 # TCP_KEEPCNT;
...

Securing PostgreSQL ClusterDB - pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
# host all all 127.0.0.1/32 trust
host all all 192.168.1.1/24 trust
# IPv6 local connections:
# host all all ::1/128 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
# local replication all trust
# host replication all 127.0.0.1/32 trust
# host replication all ::1/128 trust

Backup and Recovery
PostgreSQL provides the following options regarding the backup/recovery strategy:
Logical Backup
– Single database dump pg_dump
– Cluster database dump pg_dumpall
Physical Backup
– File System Level Backup
– Continuous Archiving and Point-in-Time Recovery (PITR)

Backup and Recovery - Logical Backup
Logical Backup
pg_dump create a TEXT file that can be restored by psql
Restore Dump File on database dbtest01_restore
$ pg_dump dbtest01 > /backup_dir/dbtest01_dump_20180716.dmp
$ psql --set ON_ERROR_STOP=on dbtest01_restore <
/backup_dir/dbtest01_dump_20180716.dmp

Backup and Recovery - Physical Backup
Physical Backup
Continuous Archiving and Point-in-Time Recovery (PITR)
#!/bin/bash
db_cluster_base="/u01/PosgreSQL"
db_cluster_dir="/u01/PosgreSQL/clusterTEST"
bckup_start=$(date +"%Y%m%d%H%M")
logfile="/u01/PosgreSQL/backup_dir/logs/clusterTEST_backup_$bckup_start.log"
backup_dir="/u01/PosgreSQL/backup_dir/clusterTEST"
bck_label="Start_Backup_$bckup_start“
...

Backup and Recovery - Physical Backup
...
psql postgres -L $logfile << EOF
SELECT pg_start_backup('$bck_label', false, false);
! tar zcvf $backup_dir/backup_$bckup_start.tar.gz --warning=no-file-changed
--warning=no-file-removed -C /u01/PosgreSQL clusterTEST --exclude='pg_wal/*‘
--exclude='postmaster*' --exclude='pg_replslot/*'
SELECT * FROM pg_stop_backup(false, true);
q
EOF

10/1/2018
OLTP Performance Benchmark
PostgreSQL vs Oracle
Oracle DBaaS27

OLTP Test: PostGreSQL vs Oracle
Goal
Use the same type of machine
Test the same OLTP workload on both databases
Test different CPU allocation
Compare the results

OLTP Test: PostGreSQL vs Oracle: Configuration
Server details
Main host:
• 2 * 8 cores CPU E5-2680 0 @ 2.70GHz
• OEL 7.2
• 192 GB
• Flash Storage volumes on PCIe cards (no NVMe)
VMs:
• 8 vCPU
• 8 GB RAM
Concurrent sessions:
• 100

OLTP Test: PostGreSQL vs Oracle : Configuration
Hammerdbcli PostGreSQL Test Setup
dbset db pg
diset tpcc pg_defaultdbase hammerdb
loadscript
vudestroy
vuset delay 5
vuset repeat 5
vuset showoutput 1
vuset timestamps 1
vuset logtotemp 1
vuset vu 100
vucreate
vurun

Hammerdbcli Oracle Test Setup
dbset db ora
diset connection instance jko
loadscript
vudestroy
vuset delay 5
vuset repeat 5
vuset showoutput 1
vuset timestamps 1
vuset logtotemp 1
vuset vu 100
vucreate
vurun

OLTP Test: PostGreSQL vs Oracle : Results
PostGreSQL
Time to complete the full test  4.23 mn
Average CPU Usage  88 %
Transaction per minutes max  156’222

Oracle
Time to complete the full test  Time 4.12 mn

select sum(value) from v$sysstat where name = 'user commits' or name = 'user rollbacks'
select sum(xact_commit + xact_rollback) from pg_stat_database

Lets Scale!
Main host:
• 2 * 8 cores CPU E5-2680 0 @ 2.70GHz
• OEL 7.2
• 192 GB
• Flash Storage volumes on PCIe cards (no NVMe)
VMs:
• 16 vCPU
• 8 GB RAM
Concurrent sessions:
• 100

PostGreSQL

Oracle

select sum(value) from v$sysstat where name = 'user commits' or name = 'user rollbacks'
select sum(xact_commit + xact_rollback) from pg_stat_database

8 vCPU
2.6% Faster 16% Less CPU 9.3% More TPM

16 vCPU
3.4% Faster 12.3% Less CPU 22.43% More TPM

41 10/1/2018
Conclusion
Oracle DBaaS

Conclusions
 PostgreSQL is a mature, enterprise RDBMS
 PostgreSQL offers good performances
 PostgreSQL offers High Availability solution

Emilian Fusaglia, Principal Consultant
Tel. +41-79-909 7213 Emiliano.Fusaglia@trivadis.com
Jacques Kostic, Principal Consultant
Tel. +41-79-909 7263 Jacques.Kostic@trivadis.com
10/1/201843 TechEvent September 2018

Session Feedback – now
TechEvent September 201844 14.09.2018
Please use the Trivadis Events mobile app to give feedback on each session
Use "My schedule" if you have registered for a session
Otherwise use "Agenda" and the search function
If the mobile app does not work (or if you have a Windows smartphone), use your
smartphone browser
– URL: http://trivadis.quickmobileplatform.eu/
– User name: <your_loginname> (such as "svv")
– Password: sent by e-mail...

TechEvent PostgreSQL Best Practices

Recommended

More Related Content

What's hot (20)

Similar to TechEvent PostgreSQL Best Practices (20)

More from Trivadis (20)

Recently uploaded (20)

TechEvent PostgreSQL Best Practices