EDB® COMPARATIVE ANALYSIS

A Discussion of
High Availability
Options for
Postgres in
Containers
A Comparison of EDB Postgres Failover Manager, Patroni, and Stolon

BY: DAVE PAGE 

VICE PRESIDENT, CHIEF ARCHITECT, TOOLS & INSTALLERS

EnterpriseDB | www.enterprisedb.com
 STNETNOC

03 Introduction

04 Origins

05 Features

14 Conclusion

Comparison of EDB Postgres Failover Manager, Patroni, and Stolon as of January 25, 2019 based on
EFM: 3.3, EDB Containers: 2.3, Stolon: 0.13.0, and Patroni: 1.5.4

Published April 2019

EnterpriseDB, EDB and EDB Postgres are trademarks of EnterpriseDB Corporation.
Other names may be trademarks of their respective owners. Copyright© 2019. All rights reserved. 20190424 WWW.ENTERPRISEDB.COM
 Introduction
Clustered containers running Postgres (either PostgreSQL or EDB Postgres™
Advanced Server) require a controller to monitor and manage the cluster. This can
be entirely scripted or handled by EDB Postgres Failover Manager (EFM), Zalando’s
Patroni (a fork of Governor) or Sorint.Lab’s Stolon.

Patroni and Stolon can provide both monitoring of the cluster and management of
the Postgres instance. By contrast, EFM provides monitoring of the cluster, but
uses its script hooks to call a collection of scripts that execute the cluster
management tasks.

For the remainder of this document, the term EFM is used to refer to the
combination of EDB Failover Manager and the supporting scripts deployed in
containers and orchestrated using Kubernetes.

This document aims to compare EFM, Patroni, and Stolon and the features they
offer. It will not discuss features which are broadly comparable in all three products
or those that are not of significant benefit to container deployment.

EDB® COMPARATIVE ANALYSIS / PAGE 3 WWW.ENTERPRISEDB.COM

Origins

EFM is primarily designed as a comprehensive cluster management tool for Postgres. Its
main focus is the security and safety of the cluster. Features to support cluster management
(initialisation of new nodes, etc.) were added later through the use of hooks that call scripts,
as this type of functionality is straightforward both in the requirements and implementation.

Patroni was created by Zalando as a fork of Governor. Its aim is to manage PostgreSQL
high availability (HA) clusters using ZooKeeper, Consul or etcd, or — in more recent versions
— the native Kubernetes store. It is designed as a template for replication, not a one-size-
fits-all solution.

Stolon was created by Sorint.Lab for managing PostgreSQL HA clusters in virtual and non-
virtual environments.

EDB® COMPARATIVE ANALYSIS / PAGE 4 WWW.ENTERPRISEDB.COM

 Features
The following subsections look at various classes of features available in each product.

Failover

Both EFM and Patroni support failover and switchover, known as controlled failover. Stolon
does not support switchover except with synchronous replication. This is discussed at
https://github.com/sorintlab/stolon/blob/master/doc/manual_switchover.md and
https://github.com/sorintlab/stolon/issues/235.

Lack of switchover support is considered a major shortcoming of Stolon.

Cluster Coordination
An HA management product in a containerised environment needs to be able to discover
the nodes that are participating in the cluster and then coordinate operations within that
cluster. These two tasks may be achieved in a number of ways.

EDB® COMPARATIVE ANALYSIS / PAGE 5 WWW.ENTERPRISEDB.COM

EFM only offers the use of Kubernetes-based discovery (using the jGroups plugin), and
coordination using jGroups, a group messaging system specifically designed for reliable
coordinated group communications. jGroups is deeply integrated into EFM.

Patroni and Stolon both offer multiple options for discovery and coordination. When using
options other than Kubernetes, discovery is dependent on nodes registering their existence
in the coordination data store such that other nodes can find their details. In the case of the
Kubernetes store, Kubernetes itself is able to register the node details.

Use of Kubernetes for discovery is arguably the most robust solution, as Kubernetes itself is
responsible for registering nodes in the store.

Use of a purpose-designed group communications library, as EFM uses jGroups, is arguably

the most robust solution. It doesn’t rely on the recording of data in external data stores,
which themselves have to be launched, maintained, and offer HA to be used.

Support for multiple discovery and coordination options as seen in Patroni and Stolon
increase the code complexity (and therefore the likelihood of bugs) as well as its
maintainability.

Use of a simple design for discovery and coordination, based on a tried and tested group
communications library, offers EFM advantages over the other solutions.

Connection Routing
In order to connect to the cluster, incoming connections must be routed from a fixed
endpoint to the appropriate node.

EDB® COMPARATIVE ANALYSIS / PAGE 6 WWW.ENTERPRISEDB.COM

 EFM is designed to work with pgPool as a connection pooler and load balancer in front of
the clusters it manages. This provides both pooling (the ability to have multiple clients share
a smaller number of database connections) and the ability to offload read-only transactions
to the standby nodes in the cluster.

Both Patroni and Stolon use HAProxy to route client connections to the current master,
providing neither pooling nor load balancing. Whilst it is possible to use pgPool as well, it’s
not directly supported by either Patroni or Stolon and will require careful design and
additional work to ensure the configuration of pgPool stays in sync with the cluster state.

EFM uses pgPool to route connections with awareness of the underlying protocol, thus
allowing for the advanced features of pgPool to be easily leveraged, whilst Stolon and
Patroni require such support to be built and managed separately.

Load Balancing

EFM’s by-design use of pgPool allows read load balancing across the cluster, whilst
transactions that write data are routed to the current master.

Both Patroni and Stolon use HAProxy to route client connections to the current master, or to
other nodes, which requires applications to be written so they implement their own load
balancing. It is possible to add pgPool on top of Stolon or Patroni, but this requires non-
trivial code to be written to manage the pgPool container(s) configuration.

EFM provides tried and tested integration with pgPool to offer load balancing, whilst Stolon
and Patroni either require a custom-built container or application-based load balancing.

EDB® COMPARATIVE ANALYSIS / PAGE 7 WWW.ENTERPRISEDB.COM

Connection Pooling
EFM’s by-design use of pgPool allows connection pooling to ensure efficient use of
database server resources.

Both Patroni and Stolon use HAProxy to route client connections to the current master, or to
other nodes, and provide no native pooling capabilities.

EFM provides tried and tested integration with pgPool to offer connection pooling, whilst
Stolon and Patroni require a custom container to be built.

Node Selection
When failover events occur, there may be cases in which it is desirable to control which
node is selected as the new master. Three features are considered: tracking of the WAL
replay location, the ability to prioritise the standby nodes in a preferred order, and the
ability to exclude a node entirely from the possibility of becoming the new master.

EDB® COMPARATIVE ANALYSIS / PAGE 8 WWW.ENTERPRISEDB.COM

EFM offers all capabilities, automatically tracking the WAL replay location so it can
determine the most up-to-date standby node to promote, while also offering the user
options for prioritisation and exclusion. Patroni offers only node exclusion, whilst Stolon
only tracks the WAL replay location.

EFM clearly offers the most flexibility in node selection.

Replication Modes

Postgres offers both synchronous and asynchronous replication modes, as well as the
ability to cascade replication hierarchies. Often related to cascading replication is the
ability to replicate to a second (read-only) cluster, which is typically on a different site.

All three products support both asynchronous and synchronous replication, though
synchronous replication with EFM containers has not yet been tested.

Cascading replication is supported only by Patroni. It allows the opportunity to offload

replication to some nodes from the master.

Multi-site replica clusters are not directly supported by EFM, though this would not be
particularly difficult to add if desired.

EFM is clearly lacking replication mode features compared to Stolon and Patroni. However,
asynchronous and synchronous replication, the most important features by far, are
supported.

EDB® COMPARATIVE ANALYSIS / PAGE 9 WWW.ENTERPRISEDB.COM

 Replica Initialisation
There are a number of effective ways to initialise new replicas,  depending on the situation:

pg_basebackup may be used to initialise a new replica from an existing clusternode.

Backups may be used to initialise a new replica from a backup taken with a tool such as
BART.
Existing data directories (e.g., those orphaned by failed nodes) may potentially be
reused if the cluster was deployed in Kubernetes using a StatefulSet.
pg_rewind may be used following a switchover to convert the old master into a new
standby.

EFM initialises replicas through one of the suites of scripts included with EFM in EDB
containers. At present, those scripts can utilise either pg_basebackup or a BART backup
(automatically used if available, to avoid load on the cluster) to bootstrap a replica node.
Reuse of existing data directories in a StatefulSet and use of pg_rewind (which can only be
used in limited circumstances) is a future enhancement that is under investigation.

Patroni allows configuration of command to perform replica initialisation. This uses

pg_basebackup by default, but the user could implement scripts similar to those used in the
EFM case to support the alternative options.

Stolon utilises pg_basebackup to initialise replicas, and pg_rewind to reuse an old master.

Both EFM and Patroni offer a great deal of flexibility when it comes to initialising new cluster
nodes. EFM is on its way to providing all listed options, whilst developers of a Patroni-based
container must also write their own scripts.
Failure Monitoring

EDB® COMPARATIVE ANALYSIS / PAGE 10 WWW.ENTERPRISEDB.COM

Failure Monitoring
Failure detection, and more importantly, understanding of the detected failure scenario is
extremely important in understanding how to behave when managing a cluster of servers.
Basic monitoring of the availability of each database server instance is critical, of course.
However, understanding the true cluster state requires the ability to ascertain the nature of
any failure, not just whether a failure occurred.

We consider a number of features in this section:

The ability to detect that a database server cannot be connected to or cannot process a
query.
The ability to detect a cluster node that is no longer running.
The ability to use external witnesses to help build consensus.
The ability to test connectivity to external resources in order to determine network
issues.

EFM was primarily designed to detect, understand and deal with all types of failure that
might occur in any typical environment, whether bare metal, virtual or containerised.
Regular connections are made to each database node to ensure that they are available,
and that they haven’t stopped accepting connections for some reason. It can be configured
to test the entire connection path on each check (from initial connection through to query
execution), or to reuse an existing connection a configurable number of times before
retesting the entire path, thus allowing the user to reduce load on external authentication
systems. EFM’s group communication protocol allows it to detect loss of a node and use of
additional witness nodes. Also the ability to test connectivity to a well-known address
allows the cluster to understand where the fault truly lies before taking action.

EDB® COMPARATIVE ANALYSIS / PAGE 11 WWW.ENTERPRISEDB.COM

 It’s difficult to fully understand the techniques used by Patroni and Stolon for monitoring as
it is largely undocumented and one has to resort to reading the code. As best as the author
can tell, both tools monitor whether or not the database on each node can be connected to
or not, and both monitor whether or not the container hosting each node is actually running.

Patroni, like EFM, runs as the container controller; thus if the container crashes or is
terminated, the loss of the node can be detected through the absence of the process by the
other nodes. It appears that Patroni relies on each node updating a TTL (time-to-live) value
in its data store; should the value expire, it is taken to indicate that a node is no longer
running.

Stolon takes a different approach, using a Sentinel container to monitor both aspects of
each node. This arguably adds the complication that the Sentinel could conceivably fail
independent of the database container, or even fail to detect the database container
failure.

EFM was designed to be a highly effective cluster-monitoring solution, understanding and

properly handling a myriad of different failure cases to ensure that it can take the most
appropriate action in each case. Patroni and Stolon offer far more simplistic monitoring
capabilities that could potentially lead to a higher chance of inappropriate corrective action
being taken, or to the failure to detect problems or false positives.

Conclusion
Whilst all three products are mature and capable, it is clear that EFM is the most appropriate
product to use as the controller in EDB’s containerised database server clusters:

It offers far more advanced cluster monitoring capabilities.

It offers connection pooling and load balancing without significant additional effort.
It uses cluster coordination that is both simple and based on deeply integrated tried-and-
tested technology.
It offers full control over node selection for failover/switchover.

As expected, there are a handful of areas in which either Patroni or Stolon (or both) have
features that EFM does not; however, these benefits are both few enough and minor
enough that they do not come close to outweighing the benefits.

EDB® COMPARATIVE ANALYSIS / PAGE 12 WWW.ENTERPRISEDB.COM

EnterpriseDB | www.enterprisedb.com
EnterpriseDB, EDB and EDB Postgres are trademarks of EnterpriseDB Corporation.
Other names may be trademarks of their respective owners. Copyright© 2019. All rights reserved. 20190424