+Copyright 2010-2016, 2ndQuadrant Limited. See the files COPYRIGHT and LICENSE for

+Copyright (c) 2010-2016, 2ndQuadrant Limited

+The contents of this file have been incorporated into the main README.md document.

+- When cloning a standby, why do I need to provide the connection parameters

+ for the primary server on the command line, not in the configuration file?

+

+ Cloning a standby is a one-time action; the role of the server being cloned

+ from could change, so fixing it in the configuration file would create

+ confusion. If `repmgr` needs to establish a connection to the primary

+ server, it can retrieve this from the `repl_nodes` table or if necessary

+ scan the replication cluster until it locates the active primary.

+

+- Why is there no foreign key on the `node_id` column in the `repl_events`

+ table?

+

+ Under some circumstances event notifications can be generated for servers

+ which have not yet been registered; it's also useful to retain a record

+ of events which includes servers removed from the replication cluster

+ which no longer have an entry in the `repl_nodes` table.

+3.1.1 2016-02-

+ Add '-P/--pwprompt' option for "repmgr create witness" (Ian)

+ Prevent repmgr/repmgrd running as root (Ian)

+

+3.1.0 2016-02-01

+ Add "repmgr standby switchover" command (Ian)

+ Revised README file (Ian)

+ Remove requirement for 'archive_mode' to be enabled (Ian)

+ Improve -?/--help output, showing default values if relevant (Ian)

+ Various bugfixes to command line/configuration parameter handling (Ian)

+

+# Copyright (c) 2ndQuadrant, 2010-2016

+# Get correct version numbers and install paths, depending on your postgres version

+PG_VERSION = $(shell pg_config --version | cut -d ' ' -f 2 | cut -d '.' -f 1,2)

+REPMGR_VERSION = $(shell grep REPMGR_VERSION version.h | cut -d ' ' -f 3 | cut -d '"' -f 2)

+PKGLIBDIR = $(shell pg_config --pkglibdir)

+SHAREDIR = $(shell pg_config --sharedir)

+

+ mkdir -p ./debian$(SHAREDIR)/contrib/

+ cp sql/repmgr_funcs.sql ./debian$(SHAREDIR)/contrib/

+ cp sql/uninstall_repmgr_funcs.sql ./debian$(SHAREDIR)/contrib/

+ mkdir -p ./debian$(PKGLIBDIR)/

+ cp sql/repmgr_funcs.so ./debian$(PKGLIBDIR)/

+ mv debian.deb ../postgresql-repmgr-$(PG_VERSION)_$(REPMGR_VERSION).deb

+The contents of this file have been incorporated into the main README.md document.

+`repmgr` is a suite of open-source tools to manage replication and failover

+within a cluster of PostgreSQL servers. It enhances PostgreSQL's built-in

+replication capabilities with utilities to set up standby servers, monitor

+replication, and perform administrative tasks such as failover or switchover

+operations.

+The `repmgr` suite provides two main tools:

+- `repmgr` - a command-line tool used to perform administrative tasks such as:

+ - setting up standby servers

+ - promoting a standby server to master

+ - switching over master and standby servers

+ - displaying the status of servers in the replication cluster

+- `repmgrd` is a daemon which actively monitors servers in a replication cluster

+ and performs the following tasks:

+ - monitoring and recording replication performance

+ - performing failover by detecting failure of the master and

+ promoting the most suitable standby server

+ - provide notifications about events in the cluster to a user-defined

+ script which can perform tasks such as sending alerts by email

+`repmgr` supports and enhances PostgreSQL's built-in streaming replication, which

+provides a single read/write master server and one or more read-only standbys

+containing near-real time copies of the master server's database.

+For a multi-master replication solution, please see 2ndQuadrant's BDR

+(bi-directional replication) extension.

+http://2ndquadrant.com/en-us/resources/bdr/

+

+For selective replication, e.g. of individual tables or databases from one server

+to another, please see 2ndQuadrant's pglogical extension.

+

+http://2ndquadrant.com/en-us/resources/pglogical/

+

+### Concepts

+

+This guide assumes that you are familiar with PostgreSQL administration and

+streaming replication concepts. For further details on streaming

+replication, see this link:

+

+ http://www.postgresql.org/docs/current/interactive/warm-standby.html#STREAMING-REPLICATION

+

+The following terms are used throughout the `repmgr` documentation.

+

+- `replication cluster`

+

+In the `repmgr` documentation, "replication cluster" refers to the network

+of PostgreSQL servers connected by streaming replication.

+

+- `node`

+

+A `node` is a server within a replication cluster.

+

+- `upstream node`

+

+This is the node a standby server is connected to; either the master server or in

+the case of cascading replication, another standby.

+

+- `failover`

+

+This is the action which occurs if a master server fails and a suitable standby

+is promoted as the new master. The `repmgrd` daemon supports automatic failover

+to minimise downtime.

+- `switchover`

+In certain circumstances, such as hardware or operating system maintenance,

+it's necessary to take a master server offline; in this case a controlled

+switchover is necessary, whereby a suitable standby is promoted and the

+existing master removed from the replication cluster in a controlled manner.

+The `repmgr` command line client provides this functionality.

+- `witness server`

+`repmgr` provides functionality to set up a so-called "witness server" to

+assist in determining a new master server in a failover situation with more

+than one standby. The witness server itself is not part of the replication

+cluster, although it does contain a copy of the repmgr metadata schema

+(see below).

+

+The purpose of a witness server is to provide a "casting vote" where servers

+in the replication cluster are split over more than one location. In the event

+of a loss of connectivity between locations, the presence or absence of

+the witness server will decide whether a server at that location is promoted

+to master; this is to prevent a "split-brain" situation where an isolated

+location interprets a network outage as a failure of the (remote) master and

+promotes a (local) standby.

+

+A witness server only needs to be created if `repmgrd` is in use.

+

+### repmgr user and metadata

+

+In order to effectively manage a replication cluster, `repmgr` needs to store

+information about the servers in the cluster in a dedicated database schema.

+This schema is automatically created during the first step in initialising

+a `repmgr`-controlled cluster (`repmgr master register`) and contains the

+following objects:

+

+tables:

+ - `repl_events`: records events of interest

+ - `repl_nodes`: connection and status information for each server in the

+ replication cluster

+ - `repl_monitor`: historical standby monitoring information written by `repmgrd`

+

+views:

+ - `repl_show_nodes`: based on the table `repl_nodes`, additionally showing the

+ name of the server's upstream node

+ - `repl_status`: when `repmgrd`'s monitoring is enabled, shows current monitoring

+ status for each node

+

+The `repmgr` metadata schema can be stored in an existing database or in its own

+dedicated database.

+

+A dedicated database superuser is required to own the meta-database as well as carry

+out administrative actions.

+### System requirements

+

+`repmgr` is developed and tested on Linux and OS X, but should work on any

+UNIX-like system supported by PostgreSQL itself.

+

+Current versions of `repmgr` support PostgreSQL from version 9.3. If you are

+interested in using `repmgr` on earlier versions of PostgreSQL you can download

+version 2.1 which supports PostgreSQL from version 9.1.

+

+All servers in the replication cluster must be running the same major version of

+PostgreSQL, and we recommend that they also run the same minor version.

+

+The `repmgr` tools must be installed on each server in the replication cluster.

+

+A dedicated system user for `repmgr` is *not* required; as many `repmgr` and

+`repmgrd` actions require direct access to the PostgreSQL data directory,

+it should be executed by the `postgres` user.

+

+Additionally, we recommend installing `rsync` and enabling passwordless

+`ssh` connectivity between all servers in the replication cluster.

+

+### Packages

+

+We recommend installing `repmgr` using the available packages for your

+system.

+

+- RedHat/CentOS: RPM packages for `repmgr` are available via Yum through

+ the PostgreSQL Global Development Group RPM repository ( http://yum.postgresql.org/ ).

+ You need to follow the instructions for your distribution (RedHat, CentOS,

+ Fedora, etc.) and architecture as detailed at yum.postgresql.org.

+- Debian/Ubuntu: the most recent `repmgr` packages are available from the

+ PostgreSQL Community APT repository ( http://apt.postgresql.org/ ).

+ Instructions can be found in the APT section of the PostgreSQL Wiki

+ ( https://wiki.postgresql.org/wiki/Apt ).

+

+See `PACKAGES.md` for details on building .deb and .rpm packages from the

+`repmgr` source code.

+

+

+### Source installation

+

+`repmgr` source code can be obtained directly from the project GitHub repository:

+

+ git clone https://github.com/2ndQuadrant/repmgr

+

+Release tarballs are also available:

+ http://repmgr.org/downloads.php

+

+`repmgr` is compiled in the same way as a PostgreSQL extension using the PGXS

+infrastructure, e.g.:

+

+ sudo make USE_PGXS=1 install

+

+`repmgr` can be built from source in any environment suitable for building

+PostgreSQL itself.

+

+

+### Configuration

+

+`repmgr` and `repmgrd` use a common configuration file, by default called

+`repmgr.conf` (although any name can be used if explicitly specified).

+At the very least, `repmgr.conf` must contain the connection parameters

+for the local `repmgr` database; see `repmgr configuration file` below

+for more details.

+

+The configuration file will be searched for in the following locations:

+

+- a configuration file specified by the `-f/--config-file` command line option

+- `repmgr.conf` in the local directory

+- `/etc/repmgr.conf`

+- the directory reported by `pg_config --sysconfdir`

+

+Note that if a file is explicitly specified with `-f/--config-file`, an error will

+be raised if it is not found or not readable and no attempt will be made to check

+default locations; this is to prevent `repmgr` reading the wrong file.

+

+For a full list of annotated configuration items, see the file `repmgr.conf.sample`.

+

+The following parameters in the configuration file can be overridden with

+command line options:

+- `-L/--log-level`

+- `-b/--pg_bindir`

+Setting up a simple replication cluster with repmgr

+---------------------------------------------------

+The following section will describe how to set up a basic replication cluster

+with a master and a standby server using the `repmgr` command line tool.

+It is assumed PostgreSQL is installed on both servers in the cluster,

+`rsync` is available and password-less SSH connections are possible between

+both servers.

+

+* * *

+

+> *TIP*: for testing `repmgr`, it's possible to use multiple PostgreSQL

+> instances running on different ports on the same computer, with

+> password-less SSH access to `localhost` enabled.

+

+* * *

+On the master server, a PostgreSQL instance must be initialised and running.

+The following replication settings must be included in `postgresql.conf`:

+ # Ensure WAL files contain enough information to enable read-only queries

+ # on the standby

+

+ # Enable up to 10 replication connections

+

+ # How much WAL to retain on the master to allow a temporarily

+ # Enable read-only queries on a standby

+ # (Note: this will be ignored on a master but we recommend including

+ # it anyway)

+ hot_standby = on

+* * *

+> *TIP*: rather than editing these settings in the default `postgresql.conf`

+> file, create a separate file such as `postgresql.replication.conf` and

+> include it from the end of the main configuration file with:

+> `include 'postgresql.replication.conf'`

+* * *

+Create a dedicated PostgreSQL superuser account and a database for

+the `repmgr` metadata, e.g.

+For the examples in this document, the name `repmgr` will be used for both

+user and database, but any names can be used.

+

+Ensure the `repmgr` user has appropriate permissions in `pg_hba.conf` and

+can connect in replication mode; `pg_hba.conf` should contain entries

+similar to the following:

+ local replication repmgr trust

+ host replication repmgr 127.0.0.1/32 trust

+ host replication repmgr 192.168.1.0/32 trust

+ local repmgr repmgr trust

+ host repmgr repmgr 127.0.0.1/32 trust

+ host repmgr repmgr 192.168.1.0/32 trust

+Adjust according to your network environment and authentication requirements.

+

+On the standby, do not create a PostgreSQL instance, but do ensure an empty

+directory is available for the `postgres` system user to create a data

+directory.

+

+

+### repmgr configuration file

+

+Create a `repmgr.conf` file on the master server. The file must contain at

+least the following parameters:

+- `cluster`: an arbitrary name for the replication cluster; this must be identical

+ on all nodes

+- `node`: a unique integer identifying the node

+- `node_name`: a unique string identifying the node; we recommend a name

+ specific to the server (e.g. 'server_1'); avoid names indicating the

+ current replication role like 'master' or 'standby' as the server's

+ role could change.

+- `conninfo`: a valid connection string for the `repmgr` database on the

+ *current* server. (On the standby, the database will not yet exist, but

+ `repmgr` needs to know the connection details to complete the setup

+ process).

+`repmgr.conf` should not be stored inside the PostgreSQL data directory,

+as it could be overwritten when setting up or reinitialising the PostgreSQL

+server. See section `Configuration` above for further details about `repmgr.conf`.

+`repmgr` will create a schema named after the cluster and prefixed with `repmgr_`,

+e.g. `repmgr_test`; we also recommend that you set the `repmgr` user's search path

+to include this schema name, e.g.

+ ALTER USER repmgr SET search_path TO repmgr_test, "$user", public;

+### Initialise the master server

+To enable `repmgr` to support a replication cluster, the master node must

+be registered with `repmgr`, which creates the `repmgr` database and adds

+a metadata record for the server:

+ $ repmgr -f repmgr.conf master register

+ [2016-01-07 16:56:46] [NOTICE] master node correctly registered for cluster test with id 1 (conninfo: host=repmgr_node1 user=repmgr dbname=repmgr)

+The metadata record looks like this:

+ repmgr=# SELECT * FROM repmgr_test.repl_nodes;

+ id | type | upstream_node_id | cluster | name | conninfo | slot_name | priority | active

+ ----+---------+------------------+---------+-------+---------------------------------------------+-----------+----------+--------

+ 1 | master | | test | node1 | host=repmgr_node1 dbname=repmgr user=repmgr | | 100 | t

+ (1 row)

+Each server in the replication cluster will have its own record and will be updated

+when its status or role changes.

+

+### Clone the standby server

+

+Create a `repmgr.conf` file on the standby server. It must contain at

+least the same parameters as the master's `repmgr.conf`, but with

+the values `node`, `node_name` and `conninfo` adjusted accordingly, e.g.:

+

+ cluster=test

+ node=2

+ node_name=node2

+ conninfo='host=repmgr_node2 user=repmgr dbname=repmgr'

+

+Clone the standby with:

+

+ $ repmgr -h repmgr_node1 -U repmgr -d repmgr -D /path/to/node2/data/ -f /etc/repmgr.conf standby clone

+ [2016-01-07 17:21:26] [NOTICE] destination directory '/path/to/node2/data/' provided

+ [2016-01-07 17:21:26] [NOTICE] starting backup...

+ [2016-01-07 17:21:26] [HINT] this may take some time; consider using the -c/--fast-checkpoint option

+ NOTICE: pg_stop_backup complete, all required WAL segments have been archived

+ [2016-01-07 17:21:28] [NOTICE] standby clone (using pg_basebackup) complete

+ [2016-01-07 17:21:28] [NOTICE] you can now start your PostgreSQL server

+ [2016-01-07 17:21:28] [HINT] for example : pg_ctl -D /path/to/node2/data/ start

+

+This will clone the PostgreSQL data directory files from the master at repmgr_node1

+using PostgreSQL's pg_basebackup utility. A `recovery.conf` file containing the

+correct parameters to start streaming from this master server will be created

+automatically, and unless otherwise the `postgresql.conf` and `pg_hba.conf`

+files will be copied from the master.

+

+Make any adjustments to the PostgreSQL configuration files now, then start the

+standby server.

+

+* * *

+

+> *NOTE*: `repmgr standby clone` does not require `repmgr.conf`, however we

+> recommend providing this as `repmgr` will set the `application_name` parameter

+> in `recovery.conf` as the value provided in `node_name`, making it easier to

+> identify the node in `pg_stat_replication`. It's also possible to provide some

+> advanced options for controlling the standby cloning process; see next section

+> for details.

+

+* * *

+

+### Verify replication is functioning

+

+Connect to the master server and execute:

+

+ repmgr=# SELECT * FROM pg_stat_replication;

+ -[ RECORD 1 ]----+------------------------------

+ pid | 7704

+ usesysid | 16384

+ usename | repmgr

+ application_name | node2

+ client_addr | 192.168.1.2

+ client_hostname |

+ client_port | 46196

+ backend_start | 2016-01-07 17:32:58.322373+09

+ backend_xmin |

+ state | streaming

+ sent_location | 0/3000220

+ write_location | 0/3000220

+ flush_location | 0/3000220

+ replay_location | 0/3000220

+ sync_priority | 0

+ sync_state | async

+

+

+### Register the standby

+

+Register the standby server with:

+

+ repmgr -f /etc/repmgr.conf standby register

+ [2016-01-08 11:13:16] [NOTICE] standby node correctly registered for cluster test with id 2 (conninfo: host=repmgr_node2 user=repmgr dbname=repmgr)

+

+Connect to the standby server's `repmgr` database and check the `repl_nodes`

+table:

+

+ repmgr=# SELECT * FROM repmgr_test.repl_nodes ORDER BY id;

+ id | type | upstream_node_id | cluster | name | conninfo | slot_name | priority | active

+ ----+---------+------------------+---------+-------+---------------------------------------------+-----------+----------+--------

+ 1 | master | | test | node1 | host=repmgr_node1 dbname=repmgr user=repmgr | | 100 | t

+ 2 | standby | 1 | test | node2 | host=repmgr_node2 dbname=repmgr user=repmgr | | 100 | t

+ (2 rows)

+

+The standby server now has a copy of the records for all servers in the

+replication cluster. Note that the relationship between master and standby is

+explicitly defined via the `upstream_node_id` value, which shows here that the

+standby's upstream server is the replication cluster master. While of limited

+use in a simple master/standby replication cluster, this information is required

+to effectively manage cascading replication (see below).

+

+

+Advanced options for cloning a standby

+--------------------------------------

+

+The above section demonstrates the simplest possible way to cloneb a standby

+server. Depending on your circumstances, finer-grained controlover the cloning

+process may be necessary.

+

+### pg_basebackup options when cloning a standby

+

+By default, `pg_basebackup` performs a checkpoint before beginning the backup

+process. However, a normal checkpoint may take some time to complete;

+a fast checkpoint can be forced with the `-c/--fast-checkpoint` option.

+However this may impact performance of the server being cloned from

+so should be used with care.

+

+Further options can be passed to the `pg_basebackup` utility via

+the setting `pg_basebackup_options` in `repmgr.conf`. See the PostgreSQL

+documentation for more details of available options:

+ http://www.postgresql.org/docs/current/static/app-pgbasebackup.html

+

+### Using rsync to clone a standby

+

+By default `repmgr` uses the `pg_basebackup` utility to clone a standby's

+data directory from the master. Under some circumstances it may be

+desirable to use `rsync` to do this, such as when resyncing the data

+directory of a failed server with an active replication node.

+

+To use `rsync` instead of `pg_basebackup`, provide the `-r/--rsync-only`

+option when executing `repmgr standby clone`.

+

+Note that `repmgr` forces `rsync` to use `--checksum` mode to ensure that all

+the required files are copied. This results in additional I/O on both source

+and destination server as the contents of files existing on both servers need

+to be compared, meaning this method is not necessarily faster than making a

+fresh clone with `pg_basebackup`.

+

+

+### Dealing with PostgreSQL configuration files

+

+By default, `repmgr` will attempt to copy the standard configuration files

+(`postgresql.conf`, `pg_hba.conf` and `pg_ident.conf`) even if they are located

+outside of the data directory (though currently they will be copied

+into the standby's data directory). To prevent this happening, when executing

+`repmgr standby clone` provide the `--ignore-external-config-files` option.

+

+If using `rsync` to clone a standby, additional control over which files

+not to transfer is possible by configuring `rsync_options` in `repmgr.conf`,

+which enables any valid `rsync` options to be passed to that command, e.g.:

+

+ rsync_options='--exclude=postgresql.local.conf'

+

+

+Setting up cascading replication with repmgr

+--------------------------------------------

+

+Cascading replication, introduced with PostgreSQL 9.2, enables a standby server

+to replicate from another standby server rather than directly from the master,

+meaning replication changes "cascade" down through a hierarchy of servers. This

+can be used to reduce load on the master and minimize bandwith usage between

+sites.

+

+`repmgr` supports cascading replication. When cloning a standby, in `repmgr.conf`

+set the parameter `upstream_node` to the id of the server the standby

+should connect to, and `repmgr` will perform the clone using this server

+and create `recovery.conf` to point to it. Note that if `upstream_node`

+is not explicitly provided, `repmgr` will use the master as the server

+to clone from.

+

+To demonstrate cascading replication, ensure you have a master and standby

+set up as shown above in the section "Setting up a simple replication cluster

+with repmgr". Create an additional standby server with `repmgr.conf` looking

+like this:

+

+ cluster=test

+ node=3

+ node_name=node3

+ conninfo='host=repmgr_node3 user=repmgr dbname=repmgr'

+ upstream_node=2

+

+Ensure `upstream_node` contains the `node` id of the previously

+created standby. Clone this standby (using the connection parameters

+for the existing standby) and register it:

+

+ $ repmgr -h repmgr_node2 -U repmgr -d repmgr -D /path/to/node3/data/ -f /etc/repmgr.conf standby clone

+ [2016-01-08 13:44:52] [NOTICE] destination directory 'node_3/data/' provided

+ [2016-01-08 13:44:52] [NOTICE] starting backup (using pg_basebackup)...

+ [2016-01-08 13:44:52] [HINT] this may take some time; consider using the -c/--fast-checkpoint option

+ [2016-01-08 13:44:52] [NOTICE] standby clone (using pg_basebackup) complete

+ [2016-01-08 13:44:52] [NOTICE] you can now start your PostgreSQL server

+ [2016-01-08 13:44:52] [HINT] for example : pg_ctl -D /path/to/node_3/data start

+

+ $ repmgr -f /etc/repmgr.conf standby register

+ [2016-01-08 14:04:32] [NOTICE] standby node correctly registered for cluster test with id 3 (conninfo: host=repmgr_node3 dbname=repmgr user=repmgr)

+

+After starting the standby, the `repl_nodes` table will look like this:

+

+ repmgr=# SELECT * FROM repmgr_test.repl_nodes ORDER BY id;

+ id | type | upstream_node_id | cluster | name | conninfo | slot_name | priority | active

+ ----+---------+------------------+---------+-------+---------------------------------------------+-----------+----------+--------

+ 1 | master | | test | node1 | host=repmgr_node1 dbname=repmgr user=repmgr | | 100 | t

+ 2 | standby | 1 | test | node2 | host=repmgr_node2 dbname=repmgr user=repmgr | | 100 | t

+ 3 | standby | 2 | test | node3 | host=repmgr_node3 dbname=repmgr user=repmgr | | 100 | t

+ (3 rows)

+

+

+Using replication slots with repmgr

+-----------------------------------

+

+Replication slots were introduced with PostgreSQL 9.4 and are designed to ensure

+that any standby connected to the master using a replication slot will always

+be able to retrieve the required WAL files. This removes the need to manually

+manage WAL file retention by estimating the number of WAL files that need to

+be maintained on the master using `wal_keep_segments`. Do however be aware

+that if a standby is disconnected, WAL will continue to accumulate on the master

+until either the standby reconnects or the replication slot is dropped.

+

+To enable `repmgr` to use replication slots, set the boolean parameter

+`use_replication_slots` in `repmgr.conf`:

+

+ use_replication_slots=1

+

+Note that `repmgr` will fail with an error if this option is specified when

+working with PostgreSQL 9.3.

+

+When cloning a standby, `repmgr` will automatically generate an appropriate

+slot name, which is stored in the `repl_nodes` table, and create the slot

+on the master:

+

+ repmgr=# SELECT * from repl_nodes ORDER BY id;

+ id | type | upstream_node_id | cluster | name | conninfo | slot_name | priority | active

+ ----+---------+------------------+---------+-------+------------------------------------------+---------------+----------+--------

+ 1 | master | | test | node1 | host=localhost dbname=repmgr user=repmgr | repmgr_slot_1 | 100 | t

+ 2 | standby | 1 | test | node2 | host=localhost dbname=repmgr user=repmgr | repmgr_slot_2 | 100 | t

+ 3 | standby | 1 | test | node3 | host=localhost dbname=repmgr user=repmgr | repmgr_slot_3 | 100 | t

+

+ repmgr=# SELECT * FROM pg_replication_slots ;

+ slot_name | plugin | slot_type | datoid | database | active | active_pid | xmin | catalog_xmin | restart_lsn

+ ---------------+--------+-----------+--------+----------+--------+------------+------+--------------+-------------

+ repmgr_slot_3 | | physical | | | t | 26060 | | | 0/50028F0

+ repmgr_slot_2 | | physical | | | t | 26079 | | | 0/50028F0

+ (2 rows)

+

+Note that a slot name will be created by default for the master but not

+actually used unless the master is converted to a standby using e.g.

+`repmgr standby switchover`.

+

+Be aware that when initially cloning a standby, you will need to ensure

+that all required WAL files remain available while the cloning is taking

+place. If using the default `pg_basebackup` method, we recommend setting

+`pg_basebackup`'s `--xlog-method` parameter to `stream` like this:

+

+ pg_basebackup_options='--xlog-method=stream'

+

+See the `pg_basebackup` documentation for details:

+ http://www.postgresql.org/docs/current/static/app-pgbasebackup.html

+

+Otherwise it's necessary to set `wal_keep_segments` to an appropriately high

+value.

+

+Further information on replication slots in the PostgreSQL documentation:

+ http://www.postgresql.org/docs/current/interactive/warm-standby.html#STREAMING-REPLICATION-SLOTS

+

+

+Promoting a standby server with repmgr

+--------------------------------------

+

+If a master server fails or needs to be removed from the replication cluster,

+a new master server must be designated, to ensure the cluster continues

+working correctly. This can be done with `repmgr standby promote`, which promotes

+the standby on the current server to master

+To demonstrate this, set up a replication cluster with a master and two attached

+standby servers so that the `repl_nodes` table looks like this:

+ repmgr=# SELECT * FROM repmgr_test.repl_nodes ORDER BY id;

+ id | type | upstream_node_id | cluster | name | conninfo | slot_name | priority | active

+ ----+---------+------------------+---------+-------+---------------------------------------------+-----------+----------+--------

+ 1 | master | | test | node1 | host=repmgr_node1 dbname=repmgr user=repmgr | | 100 | t

+ 2 | standby | 1 | test | node2 | host=repmgr_node2 dbname=repmgr user=repmgr | | 100 | t

+ 3 | standby | 1 | test | node3 | host=repmgr_node3 dbname=repmgr user=repmgr | | 100 | t

+ (3 rows)

+

+Stop the current master with e.g.:

+

+ $ pg_ctl -D /path/to/node_1/data -m fast stop

+At this point the replication cluster will be in a partially disabled state with

+both standbys accepting read-only connections while attempting to connect to the

+stopped master. Note that the `repl_nodes` table will not yet have been updated

+and will still show the master as active.

+Promote the first standby with:

+ $ repmgr -f /etc/repmgr.conf standby promote

+This will produce output similar to the following:

+ [2016-01-08 16:07:31] [ERROR] connection to database failed: could not connect to server: Connection refused

+ Is the server running on host "repmgr_node1" (192.161.2.1) and accepting

+ TCP/IP connections on port 5432?

+ could not connect to server: Connection refused

+ Is the server running on host "repmgr_node1" (192.161.2.1) and accepting

+ TCP/IP connections on port 5432?

+ [2016-01-08 16:07:31] [NOTICE] promoting standby

+ [2016-01-08 16:07:31] [NOTICE] promoting server using '/usr/bin/postgres/pg_ctl -D /path/to/node_2/data promote'

+ server promoting

+ [2016-01-08 16:07:33] [NOTICE] STANDBY PROMOTE successful

+Note: the first `[ERROR]` is `repmgr` attempting to connect to the current

+master to verify that it has failed. If a valid master is found, `repmgr`

+will refuse to promote a standby.

+The `repl_nodes` table will now look like this:

+

+ id | type | upstream_node_id | cluster | name | conninfo | slot_name | priority | active

+ ----+---------+------------------+---------+-------+---------------------------------------------+-----------+----------+--------

+ 1 | master | | test | node1 | host=repmgr_node1 dbname=repmgr user=repmgr | | 100 | f

+ 2 | master | | test | node2 | host=repmgr_node2 dbname=repmgr user=repmgr | | 100 | t

+ 3 | standby | 1 | test | node3 | host=repmgr_node3 dbname=repmgr user=repmgr | | 100 | t

+ (3 rows)

+The previous master has been marked as inactive, and `node2`'s `upstream_node_id`

+has been cleared as it's now the "topmost" server in the replication cluster.

+However the sole remaining standby is still trying to replicate from the failed

+master; `repmgr standby follow` must now be executed to rectify this situation.

+Following a new master server with repmgr

+-----------------------------------------

+Following the failure or removal of the replication cluster's existing master

+server, `repmgr standby follow` can be used to make 'orphaned' standbys

+follow the new master and catch up to its current state.

+To demonstrate this, assuming a replication cluster in the same state as the

+end of the preceding section ("Promoting a standby server with repmgr"),

+execute this:

+

+ $ repmgr -f /etc/repmgr.conf -D /path/to/node_3/data/ -h repmgr_node2 -U repmgr -d repmgr standby follow

+ [2016-01-08 16:57:06] [NOTICE] restarting server using '/usr/bin/postgres/pg_ctl -D /path/to/node_3/data/ -w -m fast restart'

+ waiting for server to shut down.... done

+ server stopped

+ waiting for server to start.... done

+ server started

+

+The standby is now replicating from the new master and `repl_nodes` has been

+updated to reflect this:

+

+ id | type | upstream_node_id | cluster | name | conninfo | slot_name | priority | active

+ ----+---------+------------------+---------+-------+---------------------------------------------+-----------+----------+--------

+ 1 | master | | test | node1 | host=repmgr_node1 dbname=repmgr user=repmgr | | 100 | f

+ 2 | master | | test | node2 | host=repmgr_node2 dbname=repmgr user=repmgr | | 100 | t

+ 3 | standby | 2 | test | node3 | host=repmgr_node3 dbname=repmgr user=repmgr | | 100 | t

+ (3 rows)

+

+Note that with cascading replication, `repmgr standby follow` can also be

+used to detach a standby from its current upstream server and follow another

+upstream server, including the master.

+

+

+Performing a switchover with repmgr

+A typical use-case for replication is a combination of master and standby

+server, with the standby serving as a backup which can easily be activated

+in case of a problem with the master. Such an unplanned failover would

+normally be handled by promoting the standby, after which an appropriate

+action must be taken to restore the old master.

+

+In some cases however it's desirable to promote the standby in a planned

+way, e.g. so maintenance can be performed on the master; this kind of switchover

+is supported by the `repmgr standby switchover` command.

+

+`repmgr standby switchover` differs from other `repmgr` actions in that it

+also performs actions on another server, for which reason you must provide

+both passwordless SSH access and the path of `repmgr.conf` on that server.

+

+* * *

+

+> *NOTE* `repmgr standby switchover` performs a relatively complex series

+> of operations on two servers, and should therefore be performed after

+> careful preparation and with adequate attention. In particular you should

+> be confident that your network environment is stable and reliable.

+>

+> We recommend running `repmgr standby switchover` at the most verbose

+> logging level (`--log-level DEBUG --verbose`) and capturing all output

+> to assist troubleshooting any problems.

+>

+> Please also read carefully the list of caveats below.

+

+* * *

+

+To demonstrate switchover, we will assume a replication cluster running on

+PostgreSQL 9.5 or later with a master (`node1`) and a standby (`node2`);

+after the switchover `node2` should become the master with `node1` following it.

+

+The switchover command must be run from the standby which is to be promoted,

+and in its simplest form looks like this:

+

+ repmgr -f /etc/repmgr.conf -C /etc/repmgr.conf standby switchover

+

+`-f /etc/repmgr.conf` is, as usual the local `repmgr` node's configuration file.

+`-C /etc/repmgr.conf` is the path to the configuration file on the current

+master, which is required to execute `repmgr` remotely on that server;

+if it is not provided with `-C`, `repmgr` will check the same path as on the

+local server, as well as the normal default locations. `repmgr` will check

+this file can be found before performing any further actions.

+

+ $ repmgr -f /etc/repmgr.conf -C /etc/repmgr.conf standby switchover -v

+ [2016-01-27 16:38:33] [NOTICE] using configuration file "/etc/repmgr.conf"

+ [2016-01-27 16:38:33] [NOTICE] switching current node 2 to master server and demoting current master to standby...

+ [2016-01-27 16:38:34] [NOTICE] 5 files copied to /tmp/repmgr-node1-archive

+ [2016-01-27 16:38:34] [NOTICE] connection to database failed: FATAL: the database system is shutting down

+

+ [2016-01-27 16:38:34] [NOTICE] current master has been stopped

+ [2016-01-27 16:38:34] [ERROR] connection to database failed: FATAL: the database system is shutting down

+

+ [2016-01-27 16:38:34] [NOTICE] promoting standby

+ [2016-01-27 16:38:34] [NOTICE] promoting server using '/usr/local/bin/pg_ctl -D /var/lib/postgresql/9.5/node_2/data promote'

+ server promoting

+ [2016-01-27 16:38:36] [NOTICE] STANDBY PROMOTE successful

+ [2016-01-27 16:38:36] [NOTICE] Executing pg_rewind on old master server

+ [2016-01-27 16:38:36] [NOTICE] 5 files copied to /var/lib/postgresql/9.5/data

+ [2016-01-27 16:38:36] [NOTICE] restarting server using '/usr/local/bin/pg_ctl -w -D /var/lib/postgresql/9.5/node_1/data -m fast restart'

+ pg_ctl: PID file "/var/lib/postgresql/9.5/node_1/data/postmaster.pid" does not exist

+ Is server running?

+ starting server anyway

+ [2016-01-27 16:38:37] [NOTICE] node 1 is replicating in state "streaming"

+ [2016-01-27 16:38:37] [NOTICE] switchover was successful

+

+Messages containing the line `connection to database failed: FATAL: the database

+system is shutting down` are not errors - `repmgr` is polling the old master database

+to make sure it has shut down correctly. `repmgr` will also archive any

+configuration files in the old master's data directory as they will otherwise

+be overwritten by `pg_rewind`; they are restored once the `pg_rewind` operation

+has completed.

+

+The old master is now replicating as a standby from the new master and `repl_nodes`

+should have been updated to reflect this:

+

+ repmgr=# SELECT * from repl_nodes ORDER BY id;

+ id | type | upstream_node_id | cluster | name | conninfo | slot_name | priority | active

+ ----+---------+------------------+---------+-------+------------------------------------------+-----------+----------+--------

+ 1 | standby | 2 | test | node1 | host=localhost dbname=repmgr user=repmgr | | 100 | t

+ 2 | master | | test | node2 | host=localhost dbname=repmgr user=repmgr | | 100 | t

+ (2 rows)

+

+

+### Caveats

+

+- the functionality provided `repmgr standby switchover` is primarily aimed

+ at a two-server master/standby replication cluster and currently does

+ not support additional standbys.

+- `repmgr standby switchover` is designed to use the `pg_rewind` utility,

+ standard in 9.5 and later and available for seperately in 9.3 and 9.4

+ (see note below)

+- `pg_rewind` *requires* that either `wal_log_hints` is enabled, or that

+ data checksums were enabled when the cluster was initialized. See the

+ `pg_rewind` documentation for details:

+ http://www.postgresql.org/docs/current/static/app-pgrewind.html

+- `repmgrd` should not be running when a switchover is carried out, otherwise

+ the `repmgrd` may try and promote a standby by itself.

+- Any other standbys attached to the old master will need to be manually

+ instructed to point to the new master (e.g. with `repmgr standby follow`).

+

+We hope to remove some of these restrictions in future versions of `repmgr`.

+

+

+### Switchover and PostgreSQL 9.3/9.4

+

+In order to efficiently reintegrate a demoted master into the replication

+cluster as a standby, it's necessary to resynchronise its data directory

+with that of the current master, as it's very likely that their timelines

+will have diverged slightly following the shutdown of the old master.

+

+The utility `pg_rewind` provides an efficient way of doing this, however

+is not included in the core PostgreSQL distribution for versions 9.3 and 9.4.

+Hoever, `pg_rewind` is available separately for these versions and we

+strongly recommend its installation. To use it with versions 9.3 and 9.4,

+provide the command line option `--pg_rewind`, optionally with the

+path to the `pg_rewind` binary location if not installed in the PostgreSQL

+`bin` directory.

+

+`pg_rewind` for versions 9.3 and 9.4 can be obtained from:

+ https://github.com/vmware/pg_rewind

+

+If `pg_rewind` is not available, as a fallback `repmgr` will use `repmgr

+standby clone` to resynchronise the old master's data directory using

+`rsync`. However, in order to ensure all files are synchronised, the

+entire data directory on both servers must be scanned, a process which

+can take some time on larger databases, in which case you should

+consider making a fresh standby clone.

+

+

+Unregistering a standby from a replication cluster

+--------------------------------------------------

+

+To unregister a running standby, execute:

+

+ repmgr standby unregister -f /etc/repmgr.conf

+

+This will remove the standby record from `repmgr`'s internal metadata

+table (`repl_nodes`). A `standby_unregister` event notification will be

+recorded in the `repl_events` table.

+

+Note that this command will not stop the server itself or remove

+it from the replication cluster.

+

+If the standby is not running, the standby record must be manually

+removed from the `repl_nodes` table with e.g.:

+

+ DELETE FROM repmgr_test.repl_nodes WHERE id = 3;

+

+Adjust schema and node ID accordingly. A future `repmgr` release

+will make it possible to unregister failed standbys.

+

+

+Automatic failover with repmgrd

+-------------------------------

+

+follow the new master.

+

+To use `repmgrd` for automatic failover, the following `repmgrd` options must

+be set in `repmgr.conf`:

+

+ failover=automatic

+ promote_command='repmgr standby promote -f /etc/repmgr/repmgr.conf'

+ follow_command='repmgr standby follow -f /etc/repmgr/repmgr.conf'

+(See `repmgr.conf.sample` for further `repmgrd`-specific settings).

+When `failover` is set to `automatic`, upon detecting failure of the current

+master, `repmgrd` will execute one of `promote_command` or `follow_command`,

+depending on whether the current server is becoming the new master or

+needs to follow another server which has become the new master. Note that

+these commands can be any valid shell script which results in one of these

+actions happening, but we strongly recommend executing `repmgr` directly.

+`repmgrd` can be started simply with e.g.:

+ repmgrd -f /etc/repmgr.conf --verbose > $HOME/repmgr/repmgr.log 2>&1

+Note that currently `repmgrd` is not required to run on the master server.

+

+To demonstrate automatic failover, set up a 3-node replication cluster (one master

+and two standbys streaming directly from the master) so that the `repl_nodes`

+table looks like this:

+

+ repmgr=# SELECT * FROM repmgr_test.repl_nodes ORDER BY id;

+ id | type | upstream_node_id | cluster | name | conninfo | slot_name | priority | active

+ ----+---------+------------------+---------+-------+---------------------------------------------+-----------+----------+--------

+ 1 | master | | test | node1 | host=repmgr_node1 dbname=repmgr user=repmgr | | 100 | t

+ 2 | standby | 1 | test | node2 | host=repmgr_node2 dbname=repmgr user=repmgr | | 100 | t

+ 3 | standby | 1 | test | node3 | host=repmgr_node3 dbname=repmgr user=repmgr | | 100 | t

+ (3 rows)

+

+

+Start `repmgrd` on each standby and verify that it's running by examining

+the log output, which at default log level will look like this:

+

+ [2016-01-05 13:15:40] [INFO] checking cluster configuration with schema 'repmgr_test'

+ [2016-01-05 13:15:40] [INFO] checking node 2 in cluster 'test'

+ [2016-01-05 13:15:40] [INFO] reloading configuration file and updating repmgr tables

+ [2016-01-05 13:15:40] [INFO] starting continuous standby node monitoring

+

+Each `repmgrd` should also have noted its successful startup in the `repl_events`

+table:

+

+ repmgr=# SELECT * FROM repl_events WHERE event = 'repmgrd_start';

+ node_id | event | successful | event_timestamp | details

+ ---------+---------------+------------+-------------------------------+---------

+ 2 | repmgrd_start | t | 2016-01-27 18:22:38.080231+09 |

+ 3 | repmgrd_start | t | 2016-01-27 18:22:38.08756+09 |

+ (2 rows)

+

+Now stop the current master server with e.g.:

+

+ pg_ctl -D /path/to/node1/data -m immediate stop

+

+This will force the master node to shut down straight away, aborting all

+processes and transactions. This will cause a flurry of activity in

+the `repmgrd` log files as each `repmgrd` detects the failure of the master

+and a failover decision is made. Here extracts from the standby server

+promoted to new master:

+

+ [2016-01-06 18:32:58] [WARNING] connection to upstream has been lost, trying to recover... 15 seconds before failover decision

+ [2016-01-06 18:33:03] [WARNING] connection to upstream has been lost, trying to recover... 10 seconds before failover decision

+ [2016-01-06 18:33:08] [WARNING] connection to upstream has been lost, trying to recover... 5 seconds before failover decision

+ ...

+ [2016-01-06 18:33:18] [NOTICE] this node is the best candidate to be the new master, promoting...

+ ...

+ [2016-01-06 18:33:20] [NOTICE] STANDBY PROMOTE successful

+

+and here from the standby server which is now following the new master:

+

+ [2016-01-06 18:32:58] [WARNING] connection to upstream has been lost, trying to recover... 15 seconds before failover decision

+ [2016-01-06 18:33:03] [WARNING] connection to upstream has been lost, trying to recover... 10 seconds before failover decision

+ [2016-01-06 18:33:08] [WARNING] connection to upstream has been lost, trying to recover... 5 seconds before failover decision

+ ...

+ [2016-01-06 18:33:23] [NOTICE] node 2 is the best candidate for new master, attempting to follow...

+ [2016-01-06 18:33:23] [INFO] changing standby's master

+ ...

+ [2016-01-06 18:33:25] [NOTICE] node 3 now following new upstream node 2

+

+The `repl_nodes` table should have been updated to reflect the new situation,

+with the original master (`node1`) marked as inactive, and standby `node3`

+now following the new master (`node2`):

+

+ repmgr=# SELECT * from repl_nodes ORDER BY id;

+ id | type | upstream_node_id | cluster | name | conninfo | slot_name | priority | active

+ ----+---------+------------------+---------+-------+------------------------------------------+-----------+----------+--------

+ 1 | master | | test | node1 | host=localhost dbname=repmgr user=repmgr | | 100 | f

+ 2 | master | | test | node2 | host=localhost dbname=repmgr user=repmgr | | 100 | t

+ 3 | standby | 2 | test | node3 | host=localhost dbname=repmgr user=repmgr | | 100 | t

+ (3 rows)

+The `repl_events` table will contain a summary of what happened to each server

+during the failover:

+

+ repmgr=# SELECT * from repmgr_test.repl_events where event_timestamp>='2016-01-06 18:30';

+ node_id | event | successful | event_timestamp | details

+ ---------+--------------------------+------------+-------------------------------+----------------------------------------------------------

+ 2 | standby_promote | t | 2016-01-06 18:33:20.061736+09 | node 2 was successfully promoted to master

+ 2 | repmgrd_failover_promote | t | 2016-01-06 18:33:20.067132+09 | node 2 promoted to master; old master 1 marked as failed

+ 3 | repmgrd_failover_follow | t | 2016-01-06 18:33:25.331012+09 | node 3 now following new upstream node 2

+ (3 rows)

+

+

+repmgrd log rotation

+--------------------

+ /var/log/postgresql/repmgr-9.5.log {

+ last_monitor_time | 2016-01-05 14:02:34.51713+09

+The interval in which monitoring history is written is controlled by the

+configuration parameter `monitor_interval_secs`; default is 2.

+As this can generate a large amount of monitoring data in the `repl_monitor`

+table , it's advisable to regularly purge historical data with

+`repmgr cluster cleanup`; use the `-k/--keep-history` to specify how

+many day's worth of data should be retained.

+Using a witness server with repmgrd

+------------------------------------

+In a situation caused e.g. by a network interruption between two

+data centres, it's important to avoid a "split-brain" situation where

+both sides of the network assume they are the active segment and the

+side without an active master unilaterally promotes one of its standbys.

+To prevent this situation happening, it's essential to ensure that one

+network segment has a "voting majority", so other segments will know

+they're in the minority and not attempt to promote a new master. Where

+an odd number of servers exists, this is not an issue. However, if each

+network has an even number of nodes, it's necessary to provide some way

+of ensuring a majority, which is where the witness server becomes useful.

+This is not a fully-fledged standby node and is not integrated into

+replication, but it effectively represents the "casting vote" when

+deciding which network segment has a majority. A witness server can

+be set up using `repmgr witness create` (see below for details) and

+can run on a dedicated server or an existing node. Note that it only

+makes sense to create a witness server in conjunction with running

+`repmgrd`; the witness server will require its own `repmgrd` instance.

+repmgrd and cascading replication

+---------------------------------

+Generating event notifications with repmgr/repmgrd

+--------------------------------------------------

+Each time `repmgr` or `repmgrd` perform a significant event, a record

+of that event is written into the `repl_events` table together with

+a timestamp, an indication of failure or success, and further details

+if appropriate. This is useful for gaining an overview of events

+affecting the replication cluster. However note that this table has

+advisory character and should be used in combination with the `repmgr`

+and PostgreSQL logs to obtain details of any events.

+Example output after a master was registered and a standby cloned

+and registered:

+ repmgr=# SELECT * from repmgr_test.repl_events ;

+ node_id | event | successful | event_timestamp | details

+ ---------+------------------+------------+-------------------------------+-------------------------------------------------------------------------------------

+ 1 | master_register | t | 2016-01-08 15:04:39.781733+09 |

+ 2 | standby_clone | t | 2016-01-08 15:04:49.530001+09 | Cloned from host 'repmgr_node1', port 5432; backup method: pg_basebackup; --force: N

+ 2 | standby_register | t | 2016-01-08 15:04:50.621292+09 |

+ (3 rows)

+Additionally, event notifications can be passed to a user-defined program

+or script which can take further action, e.g. send email notifications.

+This is done by setting the `event_notification_command` parameter in

+`repmgr.conf`.

+This parameter accepts the following format placeholders:

+ %n - node ID

+ %e - event type

+ %s - success (1 or 0)

+ %t - timestamp

+ %d - details

+The values provided for "%t" and "%d" will probably contain spaces,

+so should be quoted in the provided command configuration, e.g.:

+ event_notification_command='/path/to/some/script %n %e %s "%t" "%d"'

+By default, all notifications will be passed; the notification types

+can be filtered to explicitly named ones:

+ event_notifications=master_register,standby_register,witness_create

+The following event types are available:

+ * `master_register`

+ * `standby_register`

+ * `standby_unregister`

+ * `standby_clone`

+ * `standby_promote`

+ * `standby_follow`

+ * `standby_switchover`

+ * `witness_create`

+ * `witness_create`

+ * `repmgrd_start`

+ * `repmgrd_shutdown`

+ * `repmgrd_failover_promote`

+ * `repmgrd_failover_follow`

+Note that under some circumstances (e.g. no replication cluster master could

+be located), it will not be possible to write an entry into the `repl_events`

+table, in which case `event_notification_command` can serve as a fallback.

+Upgrading repmgr

+----------------

+`repmgr` is updated regularly with point releases (e.g. 3.0.2 to 3.0.3)

+containing bugfixes and other minor improvements. Any substantial new

+functionality will be included in a feature release (e.g. 3.0.x to 3.1.x).

+

+In general `repmgr` can be upgraded as-is without any further action required,

+however feature releases may require the `repmgr` database to be upgraded.

+An SQL script will be provided - please check the release notes for details.

+### Default values

+

+For some command line and most configuration file parameters, `repmgr` falls

+back to default values if values for these are not explicitly provided.

+The file `repmgr.conf.sample` documents the default value of configuration

+parameters if one is set. Of particular note is the log level, which

+defaults to NOTICE; particularly when using repmgr from the command line

+it may be useful to set this to a higher level with `-L/--log-level`. e.g.

+to `INFO`.

+

+Execute `repmgr --help` to see the default values for various command

+line parameters, particularly database connection parameters.

+

+See the section `Configuration` above for information on how the

+configuration file is located if `-f/--config-file` is not supplied.

+

+### repmgr commands

+

+The `repmgr` command line tool accepts commands for specific servers in the

+replication in the format "`server_type` `action`", or for the entire

+replication cluster in the format "`cluster` `action`". Each command is

+described below.

+

+In general, each command needs to be provided with the path to `repmgr.conf`,

+which contains connection details for the local database.

+ replication, just removes the standby's entry from the `repl_nodes` table.

+* `standby switchover`

+

+ Promotes a standby to master and demotes the existing master to a standby.

+ This command must be run on the standby to be promoted, and requires a

+ password-less SSH connection to the current master. Additionally the

+ location of the master's `repmgr.conf` file must be provided with

+ `-C/--remote-config-file`.

+

+ `repmgrd` should not be active if a switchover is attempted. This

+ restriction may be lifted in a later version.

+

+* `standby follow`

+

+ Attaches the standby to a new master. This command requires a valid

+ `repmgr.conf` file for the standby, either specified explicitly with

+ `-f/--config-file` or located in the current working directory; no

+ additional arguments are required.

+

+ This command will force a restart of the standby server. It can only be used

+ to attach a standby to a new master node.

+

+ is in use; see section "Using a witness server" above.

+

+ This command requires a `repmgr.conf` file containing a valid conninfo

+ string for the server to be created, as well as the other minimum required

+ parameters detailed in the section `repmgr configuration file` above.

+ on a server running an existing node. To use a different port, supply

+ this explicitly in the `repmgr.conf` conninfo string.

+ This command also requires the location of the witness server's data

+ directory to be provided (`-D/--datadir`) as well as valid connection

+ parameters for the master server.

+ By default this command will create a superuser and a repmgr user.

+ The `repmgr` user name will be extracted from the `conninfo` string

+ in `repmgr.conf`.

+ Displays information about each active node in the replication cluster. This

+ witness) or `FAILED` if the node doesn't respond. It polls each server

+ This command requires a valid `repmgr.conf` file to be provided; no

+ additional arguments are required.

+ $ repmgr -f /etc/repmgr.conf cluster show

+ Role | Name | Upstream | Connection String

+ ----------+-------|----------|--------------------------------------------

+ * master | node1 | | host=repmgr_node1 dbname=repmgr user=repmgr

+ standby | node2 | node1 | host=repmgr_node1 dbname=repmgr user=repmgr

+ standby | node3 | node2 | host=repmgr_node1 dbname=repmgr user=repmgr

+2ndQuadrant provides 24x7 production support for `repmgr`, including

+There is a mailing list/forum to discuss contributions or issues:

+

+* http://groups.google.com/group/repmgr

+Please report bugs and other issues to:

+

+* https://github.com/2ndQuadrant/repmgr

+

+* Take into account the fact that a standby can obtain WAL from an archive,

+ so even if direct streaming replication is interrupted, it may be up-to-date

+

+ * Copyright (C) 2ndQuadrant, 2010-2016

+ * Copyright (c) 2ndQuadrant, 2010-2016

+ * Copyright (C) 2ndQuadrant, 2010-2016

+bool config_file_found = false;

+ log_notice(_("looking for configuration file in %s\n"), sysconf_etc_path);

+ char *conninfo_errmsg = NULL;

+ bool node_found = false;

+

+ log_verbose(LOG_NOTICE, _("no configuration file provided and no default file found - "

+ {

+ options->node = repmgr_atoi(value, "node", &config_errors, false);

+ node_found = true;

+ }

+ options->upstream_node = repmgr_atoi(value, "upstream_node", &config_errors, false);

+ options->priority = repmgr_atoi(value, "priority", &config_errors, true);

+ options->master_response_timeout = repmgr_atoi(value, "master_response_timeout", &config_errors, false);

+ options->master_response_timeout = repmgr_atoi(value, "primary_response_timeout", &config_errors, false);

+ options->reconnect_attempts = repmgr_atoi(value, "reconnect_attempts", &config_errors, false);

+ options->reconnect_interval = repmgr_atoi(value, "reconnect_interval", &config_errors, false);

+ options->monitor_interval_secs = repmgr_atoi(value, "monitor_interval_secs", &config_errors, false);

+ options->retry_promote_interval_secs = repmgr_atoi(value, "retry_promote_interval_secs", &config_errors, false);

+ options->use_replication_slots = repmgr_atoi(value, "use_replication_slots", &config_errors, false);

+ if (node_found == false)

+ error_list_append(&config_errors, _("\"node\": parameter was not found"));

+ else if (options->node == 0)

+ error_list_append(&config_errors, _("\"node\": must be greater than zero"));

+ if (strlen(options->conninfo))

+repmgr_atoi(const char *value, const char *config_item, ErrorList *error_list, bool allow_negative)

+ /* Disallow negative values for most parameters */

+ if (allow_negative == false && longval < 0)

+ * Copyright (c) 2ndQuadrant, 2010-2016

+ ErrorList *error_list,

+ bool allow_negative);

+ * Copyright (C) 2ndQuadrant, 2010-2016

+#include "catalog/pg_control.h"

+

+

+_establish_db_connection(const char *conninfo, const bool exit_on_error, const bool log_notice)

+ if (log_notice)

+ {

+ log_notice(_("connection to database failed: %s\n"),

+ PQerrorMessage(conn));

+ }

+ else

+ {

+ log_err(_("connection to database failed: %s\n"),

+ PQerrorMessage(conn));

+ }

+establish_db_connection(const char *conninfo, const bool exit_on_error)

+{

+ return _establish_db_connection(conninfo, exit_on_error, false);

+}

+

+PGconn *

+test_db_connection(const char *conninfo, const bool exit_on_error)

+{

+ return _establish_db_connection(conninfo, exit_on_error, true);

+}

+

+

+PGconn *

+ log_verbose(LOG_WARNING, _("get_master_node_id(): no active primary found\n"));

+ log_verbose(LOG_DEBUG, _("get_pg_setting(): returned value is \"%s\"\n"), output);

+ * get_conninfo_value()

+ *

+ * Extract the value represented by 'keyword' in 'conninfo' and copy

+ * it to the 'output' buffer.

+ *

+ * Returns true on success, or false on failure (conninfo string could

+ * not be parsed, or provided keyword not found).

+ */

+

+bool

+get_conninfo_value(const char *conninfo, const char *keyword, char *output)

+{

+ PQconninfoOption *conninfo_options;

+ PQconninfoOption *conninfo_option;

+

+ conninfo_options = PQconninfoParse(conninfo, NULL);

+

+ if (conninfo_options == false)

+ {

+ log_err(_("Unable to parse provided conninfo string \"%s\""), conninfo);

+ return false;

+ }

+

+ for (conninfo_option = conninfo_options; conninfo_option->keyword != NULL; conninfo_option++)

+ {

+ if (strcmp(conninfo_option->keyword, keyword) == 0)

+ {

+ if (conninfo_option->val != NULL && conninfo_option->val[0] != '\0')

+ {

+ strncpy(output, conninfo_option->val, MAXLEN);

+ break;

+ }

+ }

+ }

+

+ PQconninfoFree(conninfo_options);

+

+ return true;

+}

+

+

+/*

+ /*

+ * If the caller wanted to get a copy of the connection info string, sub

+ * out the local stack pointer for the pointer passed by the caller.

+ */

+ if (master_conninfo_out != NULL)

+ remote_conninfo = master_conninfo_out;

+

+ char sqlquery[QUERY_STR_LEN];

+ int query_res;

+ PGresult *res;

+ t_replication_slot slot_info;

+ query_res = get_slot_record(conn, slot_name, &slot_info);

+ if (query_res)

+ if (strcmp(slot_info.slot_type, "physical") != 0)

+ return false;

+

+ if (slot_info.active == false)

+

+

+int

+get_slot_record(PGconn *conn, char *slot_name, t_replication_slot *record)

+{

+ char sqlquery[QUERY_STR_LEN];

+ PGresult *res;

+

+ sqlquery_snprintf(sqlquery,

+ "SELECT slot_name, slot_type, active "

+ " FROM pg_replication_slots "

+ " WHERE slot_name = '%s' ",

+ slot_name);

+

+ log_verbose(LOG_DEBUG, "get_slot_record():\n%s\n", sqlquery);

+

+ res = PQexec(conn, sqlquery);

+ if (!res || PQresultStatus(res) != PGRES_TUPLES_OK)

+ {

+ log_err(_("unable to query pg_replication_slots: %s\n"),

+ PQerrorMessage(conn));

+ PQclear(res);

+ return -1;

+ }

+

+ if (!PQntuples(res))

+ {

+ return 0;

+ }

+

+ strncpy(record->slot_name, PQgetvalue(res, 0, 0), MAXLEN);

+ strncpy(record->slot_type, PQgetvalue(res, 0, 1), MAXLEN);

+ record->active = (strcmp(PQgetvalue(res, 0, 2), "t") == 0)

+ ? true

+ : false;

+

+ PQclear(res);

+

+ return 1;

+}

+

+

+ char sqlquery[QUERY_STR_LEN];

+int

+get_node_record(PGconn *conn, char *cluster, int node_id, t_node_info *node_info)

+ PGresult *res;

+ int ntuples;

+

+ sqlquery_snprintf(

+ sqlquery,

+ "SELECT id, type, upstream_node_id, name, conninfo, slot_name, priority, active"

+ " FROM %s.repl_nodes "

+ " WHERE cluster = '%s' "

+ " AND id = %i",

+ get_repmgr_schema_quoted(conn),

+ cluster,

+ node_id);

+ res = PQexec(conn, sqlquery);

+ if (PQresultStatus(res) != PGRES_TUPLES_OK)

+ {

+ return -1;

+ }

+

+ ntuples = PQntuples(res);

+

+ if (ntuples == 0)

+ {

+ log_verbose(LOG_DEBUG, "get_node_record(): no record found for node %i\n", node_id);

+ return 0;

+ }

+

+ node_info->node_id = atoi(PQgetvalue(res, 0, 0));

+ node_info->type = parse_node_type(PQgetvalue(res, 0, 1));

+ node_info->upstream_node_id = atoi(PQgetvalue(res, 0, 2));

+ strncpy(node_info->name, PQgetvalue(res, 0, 3), MAXLEN);

+ strncpy(node_info->conninfo_str, PQgetvalue(res, 0, 4), MAXLEN);

+ strncpy(node_info->slot_name, PQgetvalue(res, 0, 5), MAXLEN);

+ node_info->priority = atoi(PQgetvalue(res, 0, 6));

+ node_info->active = (strcmp(PQgetvalue(res, 0, 7), "t") == 0)

+ ? true

+ : false;

+

+ PQclear(res);

+

+ return ntuples;

+}

+

+

+int

+get_node_replication_state(PGconn *conn, char *node_name, char *output)

+{

+ char sqlquery[QUERY_STR_LEN];

+ PGresult * res;

+

+ sqlquery_snprintf(

+ sqlquery,

+ " SELECT state "

+ " FROM pg_catalog.pg_stat_replication"

+ " WHERE application_name = '%s'",

+ node_name

+ );

+

+ res = PQexec(conn, sqlquery);

+

+ if (PQresultStatus(res) != PGRES_TUPLES_OK)

+ {

+ PQclear(res);

+ return -1;

+ }

+

+ if (PQntuples(res) == 0)

+ {

+ PQclear(res);

+ return 0;

+ }

+

+ strncpy(output, PQgetvalue(res, 0, 0), MAXLEN);

+ PQclear(res);

+

+ return true;

+

+}

+

+t_server_type

+parse_node_type(const char *type)

+{

+ if (strcmp(type, "master") == 0)

+ {

+ return MASTER;

+ }

+ else if (strcmp(type, "standby") == 0)

+ {

+ return STANDBY;

+ }

+ else if (strcmp(type, "witness") == 0)

+ {

+ return WITNESS;

+ }

+

+ return UNKNOWN;

+}

+

+

+int

+get_data_checksum_version(const char *data_directory)

+{

+ ControlFileData control_file;

+ int fd;

+ char control_file_path[MAXPGPATH];

+

+ snprintf(control_file_path, MAXPGPATH, "%s/global/pg_control", data_directory);

+ if ((fd = open(control_file_path, O_RDONLY | PG_BINARY, 0)) == -1)

+ {

+ log_err(_("Unable to open control file \"%s\" for reading: %s\n"),

+ control_file_path, strerror(errno));

+ return -1;

+ }

+

+ if (read(fd, &control_file, sizeof(ControlFileData)) != sizeof(ControlFileData))

+ {

+ log_err(_("could not read file \"%s\": %s\n"),

+ control_file_path, strerror(errno));

+ close(fd);

+ return -1;

+ }

+

+ close(fd);

+

+ return (int)control_file.data_checksum_version;

+ * Copyright (c) 2ndQuadrant, 2010-2016

+/*

+ * Struct to store replication slot information

+ */

+

+typedef struct s_replication_slot

+{

+ char slot_name[MAXLEN];

+ char slot_type[MAXLEN];

+ bool active;

+} t_replication_slot;

+

+

+PGconn *_establish_db_connection(const char *conninfo,

+ const bool exit_on_error,

+ const bool log_notice);

+ const bool exit_on_error);

+PGconn *test_db_connection(const char *conninfo,

+ const bool exit_on_error);

+bool get_conninfo_value(const char *conninfo, const char *keyword, char *output);

+int get_slot_record(PGconn *conn, char *slot_name, t_replication_slot *record);

+int get_node_record(PGconn *conn, char *cluster, int node_id, t_node_info *node_info);

+bool create_event_record(PGconn *conn, t_configuration_options *options, int node_id, char *event, bool successful, char *details);

+int get_node_replication_state(PGconn *conn, char *node_name, char *output);

+t_server_type parse_node_type(const char *type);

+int get_data_checksum_version(const char *data_directory);

+repmgr (3.1.1-1~bpo70+1) wheezy-backports-sloppy; urgency=medium

+

+ * Rebuild for wheezy-backports-sloppy.

+

+ -- Christoph Berg <[email protected]> Mon, 04 Apr 2016 11:55:30 +0200

+

+repmgr (3.1.1-1) unstable; urgency=medium

+

+ * Imported Upstream version 3.1.1

+ * Use HTTPS protocol in Vcs-Browser URI inside debian/control file

+

+ -- Marco Nenciarini <[email protected]> Tue, 23 Feb 2016 19:11:25 +0100

+

+Vcs-Browser: https://anonscm.debian.org/gitweb/?p=pkg-postgresql/repmgr.git

+Vcs-Browser: https://anonscm.debian.org/gitweb/?p=pkg-postgresql/repmgr.git

+ * Copyright (C) 2ndQuadrant, 2010-2016

+ * Copyright (C) 2ndQuadrant, 2010-2016

+ * Copyright (c) 2ndQuadrant, 2010-2016

+ * Copyright (C) 2ndQuadrant, 2010-2016

+ * STANDBY SWITCHOVER

+ * For internal use:

+ * STANDBY ARCHIVE-CONFIG

+ * STANDBY RESTORE-CONFIG

+ *

+#include <sys/types.h>

+#include <dirent.h>

+#include <sys/stat.h>

+#define STANDBY_SWITCHOVER 7

+#define STANDBY_ARCHIVE_CONFIG 8

+#define STANDBY_RESTORE_CONFIG 9

+#define WITNESS_CREATE 10

+#define CLUSTER_SHOW 11

+#define CLUSTER_CLEANUP 12

+static void do_standby_switchover(void);

+static void do_standby_archive_config(void);

+static void do_standby_restore_config(void);

+static void do_help(void);

+

+static bool remote_command(const char *host, const char *user, const char *command, PQExpBufferData *outputbuf);

+static void format_db_cli_params(const char *conninfo, char *output);

+static bool copy_file(const char *old_filename, const char *new_filename);

+bool wal_keep_segments_used = false;

+bool connection_param_provided = false;

+bool pg_rewind_supplied = false;

+ /* -l/--local-port is deprecated */

+ {"mode", required_argument, NULL, 'm'},

+ {"remote-config-file", required_argument, NULL, 'C'},

+ /* deprecated from 3.2; replaced with -P/----pwprompt */

+ {"config-archive-dir", required_argument, NULL, 5},

+ {"pg_rewind", optional_argument, NULL, 6},

+ {"pwprompt", optional_argument, NULL, 7},

+ const char *env;

+ /* Disallow running as root to prevent directory ownership problems */

+ if (geteuid() == 0)

+ {

+ fprintf(stderr,

+ _("%s: cannot be run as root\n"

+ "Please log in (using, e.g., \"su\") as the "

+ "(unprivileged) user that owns\n"

+ "the data directory.\n"

+ ),

+ progname());

+ exit(1);

+ }

+

+ /* Initialise some defaults */

+

+ /* set default user */

+ env = getenv("PGUSER");

+ if (!env)

+ {

+ struct passwd *pw = NULL;

+ pw = getpwuid(geteuid());

+ if (pw)

+ {

+ env = pw->pw_name;

+ }

+ else

+ {

+ fprintf(stderr, _("could not get current user name: %s\n"), strerror(errno));

+ exit(ERR_BAD_CONFIG);

+ }

+ }

+ strncpy(runtime_options.username, env, MAXLEN);

+

+ /* set default database */

+ env = getenv("PGDATABASE");

+ if (!env)

+ {

+ env = runtime_options.username;

+ }

+ strncpy(runtime_options.dbname, env, MAXLEN);

+

+ /* set default port */

+

+ env = getenv("PGPORT");

+ if (!env)

+ {

+ env = DEF_PGPORT_STR;

+ }

+

+ strncpy(runtime_options.masterport, env, MAXLEN);

+ while ((c = getopt_long(argc, argv, "?Vd:h:p:U:S:D:l:f:R:w:k:FWIvb:rcL:tm:C:", long_options,

+ do_help();

+ connection_param_provided = true;

+ connection_param_provided = true;

+ repmgr_atoi(optarg, "-p/--port", &cli_errors, false);

+ connection_param_provided = true;

+ connection_param_provided = true;

+ repmgr_atoi(optarg, "-l/--local-port", &cli_errors, false);

+ repmgr_atoi(optarg, "-w/--wal-keep-segments", &cli_errors, false);

+ runtime_options.keep_history = repmgr_atoi(optarg, "-k/--keep-history", &cli_errors, false);

+ termPQExpBuffer(&invalid_log_level);

+ case 'm':

+ {

+ if (strcmp(optarg, "smart") == 0 ||

+ strcmp(optarg, "fast") == 0 ||

+ strcmp(optarg, "immediate") == 0

+ )

+ {

+ strncpy(runtime_options.pg_ctl_mode, optarg, MAXLEN);

+ }

+ else

+ {

+ PQExpBufferData invalid_mode;

+ initPQExpBuffer(&invalid_mode);

+ appendPQExpBuffer(&invalid_mode, _("Invalid pg_ctl shutdown mode \"%s\" provided"), optarg);

+ error_list_append(&cli_errors, invalid_mode.data);

+ termPQExpBuffer(&invalid_mode);

+ }

+ }

+ break;

+ case 'C':

+ strncpy(runtime_options.remote_config_file, optarg, MAXLEN);

+ break;

+ case 5:

+ strncpy(runtime_options.config_archive_dir, optarg, MAXLEN);

+ break;

+ case 6:

+ if (optarg != NULL)

+ {

+ strncpy(runtime_options.pg_rewind, optarg, MAXFILENAME);

+ }

+ pg_rewind_supplied = true;

+ break;

+ case 7:

+ runtime_options.witness_pwprompt = true;

+ break;

+

+ * { MASTER | PRIMARY } REGISTER |

+ * STANDBY {REGISTER | UNREGISTER | CLONE [node] | PROMOTE | FOLLOW [node] | SWITCHOVER | REWIND} |

+ else if (strcasecmp(server_cmd, "SWITCHOVER") == 0)

+ action = STANDBY_SWITCHOVER;

+ else if (strcasecmp(server_cmd, "ARCHIVE-CONFIG") == 0)

+ action = STANDBY_ARCHIVE_CONFIG;

+ else if (strcasecmp(server_cmd, "RESTORE-CONFIG") == 0)

+ action = STANDBY_RESTORE_CONFIG;

+ /* Some configuration file items can be overriden by command line options */

+ /* Command-line parameter -L/--log-level overrides any setting in config file*/

+ if (*runtime_options.loglevel != '\0')

+ {

+ strncpy(options.loglevel, runtime_options.loglevel, MAXLEN);

+ }

+

+ case STANDBY_SWITCHOVER:

+ do_standby_switchover();

+ break;

+ case STANDBY_ARCHIVE_CONFIG:

+ do_standby_archive_config();

+ break;

+ case STANDBY_RESTORE_CONFIG:

+ do_standby_restore_config();

+ break;

+ char name_header[MAXLEN];

+ char upstream_header[MAXLEN];

+ int name_length,

+ upstream_length,

+ conninfo_length = 0;

+ "SELECT conninfo, type, name, upstream_node_name"

+ " FROM %s.repl_show_nodes",

+ get_repmgr_schema_quoted(conn));

+

+ log_verbose(LOG_DEBUG, "do_cluster_show(): \n%s\n",sqlquery );

+

+ /* Format header nicely */

+

+ strncpy(name_header, _("Name"), MAXLEN);

+ strncpy(upstream_header, _("Upstream"), MAXLEN);

+

+ /*

+ * XXX if repmgr is ever localized into non-ASCII locales,

+ * use pg_wcssize() or similar to establish printed column length

+ */

+ name_length = strlen(name_header);

+ upstream_length = strlen(upstream_header);

+

+ for (i = 0; i < PQntuples(res); i++)

+ {

+ int conninfo_length_cur, name_length_cur, upstream_length_cur;

+

+ conninfo_length_cur = strlen(PQgetvalue(res, i, 0));

+ if (conninfo_length_cur > conninfo_length)

+ conninfo_length = conninfo_length_cur;

+

+ name_length_cur = strlen(PQgetvalue(res, i, 2));

+ if (name_length_cur > name_length)

+ name_length = name_length_cur;

+

+ upstream_length_cur = strlen(PQgetvalue(res, i, 3));

+ if (upstream_length_cur > upstream_length)

+ upstream_length = upstream_length_cur;

+ }

+

+ printf("Role | %-*s | %-*s | Connection String\n", name_length, name_header, upstream_length, upstream_header);

+ printf("----------+-");

+

+ for (i = 0; i < name_length; i++)

+ printf("-");

+

+ printf("-|-");

+ for (i = 0; i < upstream_length; i++)

+ printf("-");

+

+ printf("-|-");

+ for (i = 0; i < conninfo_length; i++)

+ printf("-");

+

+ printf("\n");

+

+ printf("| %-*s ", name_length, PQgetvalue(res, i, 2));

+ printf("| %-*s ", upstream_length, PQgetvalue(res, i, 3));

+ t_node_info node_info = T_NODE_INFO_INITIALIZER;

+

+ begin_transaction(conn);

+ if (get_node_record(conn, options.cluster_name, options.node, &node_info))

+ /* XXX log registration failure? */

+ PGconn *primary_conn = NULL;

+ * If the upstream node is a standby, try to connect to the primary too so we

+ * can write an event record

+ */

+ if (is_standby(upstream_conn))

+ {

+ if (strlen(options.cluster_name))

+ {

+ primary_conn = get_master_connection(upstream_conn, options.cluster_name,

+ NULL, NULL);

+ }

+ }

+ else

+ {

+ primary_conn = upstream_conn;

+ }

+

+ /*

+ if (runtime_options.rsync_only)

+ log_notice(_("starting backup (using rsync)...\n"));

+ }

+ else

+ {

+ log_notice(_("starting backup (using pg_basebackup)...\n"));

+ if (runtime_options.fast_checkpoint == false)

+ log_hint(_("this may take some time; consider using the -c/--fast-checkpoint option\n"));

+ * If configuration files were not inside the data directory, we'll need to

+ /* Log the event - if we could connect to the primary */

+ if (primary_conn != NULL)

+ {

+ initPQExpBuffer(&event_details);

+ /* Add details about relevant runtime options used */

+ appendPQExpBuffer(&event_details,

+ _("Cloned from host '%s', port %s"),

+ runtime_options.host,

+ runtime_options.masterport);

+ appendPQExpBuffer(&event_details,

+ _("; backup method: %s"),

+ runtime_options.rsync_only ? "rsync" : "pg_basebackup");

+ appendPQExpBuffer(&event_details,

+ _("; --force: %s"),

+ runtime_options.force ? "Y" : "N");

+

+ create_event_record(primary_conn,

+ &options,

+ options.node,

+ "standby_clone",

+ true,

+ event_details.data);

+ }

+ "node %i was successfully promoted to master",

+/*

+ * Follow a new primary.

+ *

+ * This function has two "modes":

+ * 1) no primary info provided - determine primary from standby metadata

+ * 2) primary info provided - use that info to connect to the primary.

+ *

+ * (2) is mainly for when a node has been stopped as part of a switchover

+ * and needs to be started with recovery.conf correctly configured.

+ */

+

+ int master_id = 0;

+ char data_dir[MAXFILENAME];

+ log_debug("do_standby_follow()\n");

+ /*

+ * If -h/--host wasn't provided, attempt to connect to standby

+ * to determine primary, and carry out some other checks while we're

+ * at it.

+ */

+ if ( *runtime_options.host == '\0')

+ /* We need to connect to check configuration */

+ log_info(_("connecting to standby database\n"));

+ conn = establish_db_connection(options.conninfo, true);

+ log_verbose(LOG_INFO, _("connected to standby, checking its state\n"));

+ /* Check we are in a standby node */

+ retval = is_standby(conn);

+ if (retval == 0 || retval == -1)

+ {

+ log_err(_(retval == 0 ? "this command should be executed on a standby node\n" :

+ "connection to node lost!\n"));

+ PQfinish(conn);

+ exit(ERR_BAD_CONFIG);

+ }

+ /* Get the data directory full path */

+ success = get_pg_setting(conn, "data_directory", data_dir);

+

+ if (success == false)

+ log_err(_("unable to determine data directory\n"));

+ exit(ERR_BAD_CONFIG);

+ /*

+ * we also need to check if there is any master in the cluster or wait for

+ * one to appear if we have set the wait option

+ */

+ log_info(_("discovering new master...\n"));

+

+ do

+ {

+ if (!is_pgup(conn, options.master_response_timeout))

+ {

+ conn = establish_db_connection(options.conninfo, true);

+ }

+

+ master_conn = get_master_connection(conn,

+ options.cluster_name, &master_id, (char *) &master_conninfo);

+ }

+ while (master_conn == NULL && runtime_options.wait_for_master);

+

+ if (master_conn == NULL)

+ {

+ log_err(_("unable to determine new master node\n"));

+ PQfinish(conn);

+ exit(ERR_BAD_CONFIG);

+ }

+

+ /*

+ * Verify that standby and master are supported and compatible server

+ * versions

+ */

+ check_master_standby_version_match(conn, master_conn);

+ /* primary server info explictly provided - attempt to connect to that */

+ else

+ {

+ keywords[0] = "host";

+ values[0] = runtime_options.host;

+ keywords[1] = "port";

+ values[1] = runtime_options.masterport;

+

+ master_conn = establish_db_connection_by_params(keywords, values, true);

+

+ master_id = get_master_node_id(master_conn, options.cluster_name);

+

+ strncpy(data_dir, runtime_options.dest_dir, MAXFILENAME);

+ }

+

+ "standby_follow",

+ /* XXX add more detail! */

+ /*

+ * It's possible this node was an inactive primary - update the

+ * relevant fields to ensure it's marked as an active standby

+ */

+ if (update_node_record_status(master_conn,

+ options.cluster_name,

+ options.node,

+ "standby",

+ master_id,

+ true) == false)

+ {

+ log_err(_("unable to update upstream node\n"));

+

+ /* XXX add event record - possible move from repmgrd? */

+/*

+ * Perform a switchover by:

+ * - stopping current primary node

+ * - promoting this standby node to primary

+ * - forcing previous primary node to follow this node

+ *

+ * Caveats:

+ * - repmgrd must not be running, otherwise it may

+ * attempt a failover

+ * (TODO: find some way of notifying repmgrd of planned

+ * activity like this)

+ * - currently only set up for two-node operation; any other

+ * standbys will probably become downstream cascaded standbys

+ * of the old primary once it's restarted

+ * - as we're executing repmgr remotely (on the old primary),

+ * we'll need the location of its configuration file; this

+ * can be provided explicitly with -C/--remote-config-file,

+ * otherwise repmgr will look in default locations on the

+ * remote server

+ *

+ * TODO:

+ * - make connection test timeouts/intervals configurable (see below)

+ */

+

+static void

+do_standby_switchover(void)

+{

+ PGconn *local_conn;

+ PGconn *remote_conn;

+ int server_version_num;

+ bool use_pg_rewind;

+

+ /* the remote server is the primary which will be demoted */

+ char remote_conninfo[MAXCONNINFO] = "";

+ char remote_host[MAXLEN];

+ char remote_data_directory[MAXLEN];

+ int remote_node_id;

+ char remote_node_replication_state[MAXLEN] = "";

+ char remote_archive_config_dir[MAXLEN];

+ char remote_pg_rewind[MAXLEN];

+ int i,

+ r = 0;

+

+ char command[MAXLEN];

+ PQExpBufferData command_output;

+

+ char repmgr_db_cli_params[MAXLEN] = "";

+ int query_result;

+ t_node_info remote_node_record;

+ bool connection_success;

+

+

+ /*

+ * SANITY CHECKS

+ *

+ * We'll be doing a bunch of operations on the remote server (primary

+ * to be demoted) - careful checks needed before proceding.

+ */

+

+ log_notice(_("switching current node %i to master server and demoting current master to standby...\n"), options.node);

+

+ local_conn = establish_db_connection(options.conninfo, true);

+

+ /* Check that this is a standby */

+

+ if (!is_standby(local_conn))

+ {

+ log_err(_("switchover must be executed from the standby node to be promoted\n"));

+ PQfinish(local_conn);

+

+ exit(ERR_BAD_CONFIG);

+ }

+

+ server_version_num = check_server_version(local_conn, "master", true, NULL);

+

+ /*

+ * Add a friendly notice if --pg_rewind supplied for 9.5 and later - we'll

+ * be ignoring it anyway

+ */

+ if (pg_rewind_supplied == true && server_version_num >= 90500)

+ {

+ log_notice(_("--pg_rewind not required for PostgreSQL 9.5 and later\n"));

+ }

+

+ /*

+ * TODO: check that standby's upstream node is the primary

+ * (it's probably not feasible to switch over to a cascaded standby)

+ */

+

+ /* Check that primary is available */

+ remote_conn = get_master_connection(local_conn, options.cluster_name, &remote_node_id, remote_conninfo);

+

+ if (remote_conn == NULL)

+ {

+ log_err(_("unable to connect to current master node\n"));

+ log_hint(_("check that the cluster is correctly configured and this standby is registered\n"));

+ PQfinish(local_conn);

+ exit(ERR_DB_CON);

+ }

+

+ /* Get the remote's node record */

+ query_result = get_node_record(remote_conn, options.cluster_name, remote_node_id, &remote_node_record);

+

+ if (query_result < 1)

+ {

+ log_err(_("unable to retrieve node record for node %i\n"), remote_node_id);

+

+ PQfinish(local_conn);

+

+ exit(ERR_DB_QUERY);

+ }

+

+ log_debug("remote node name is \"%s\"\n", remote_node_record.name);

+

+ /*

+ * Check that we can connect by SSH to the remote (current primary) server,

+ * and read its data directory

+ *

+ * TODO: check we can read contents of PG_VERSION??

+ * -> assuming the remote user/directory is set up correctly,

+ * we should only be able to see the file as the PostgreSQL

+ * user, so it should be readable anyway

+ */

+ get_conninfo_value(remote_conninfo, "host", remote_host);

+

+ r = test_ssh_connection(remote_host, runtime_options.remote_user);

+

+ if (r != 0)

+ {

+ log_err(_("unable to connect via ssh to host %s, user %s\n"), remote_host, runtime_options.remote_user);

+ }

+

+ if (get_pg_setting(remote_conn, "data_directory", remote_data_directory) == false)

+ {

+ log_err(_("unable to retrieve master's data directory location\n"));

+ PQfinish(remote_conn);

+ PQfinish(local_conn);

+ exit(ERR_DB_CON);

+ }

+

+ log_debug("master's data directory is: %s\n", remote_data_directory);

+

+ maxlen_snprintf(command,

+ "ls %s/PG_VERSION >/dev/null 2>&1 && echo 1 || echo 0",

+ remote_data_directory);

+ initPQExpBuffer(&command_output);

+

+ (void)remote_command(

+ remote_host,

+ runtime_options.remote_user,

+ command,

+ &command_output);

+

+ if (*command_output.data == '1')

+ {

+ log_verbose(LOG_DEBUG, "PG_VERSION found in %s\n", remote_data_directory);

+ }

+ else if (*command_output.data == '0')

+ {

+ log_err(_("%s is not a PostgreSQL data directory or is not accessible to user %s\n"), remote_data_directory, runtime_options.remote_user);

+ PQfinish(remote_conn);

+ PQfinish(local_conn);

+ exit(ERR_BAD_CONFIG);

+ }

+ else

+ {

+ log_err(_("Unexpected output from remote command:\n%s\n"), command_output.data);

+ PQfinish(remote_conn);

+ PQfinish(local_conn);

+ exit(ERR_BAD_CONFIG);

+ }

+

+ termPQExpBuffer(&command_output);

+

+

+ if (server_version_num >= 90500)

+ {

+ /* 9.5 and later have pg_rewind built-in - always use that */

+ use_pg_rewind = true;

+ maxlen_snprintf(remote_pg_rewind,

+ "%s/pg_rewind",

+ pg_bindir);

+ }

+ else

+ {

+ /* 9.3/9.4 - user can use separately-compiled pg_rewind */

+ if (pg_rewind_supplied == true)

+ {

+ use_pg_rewind = true;

+

+ /* User has specified pg_rewind path */

+ if (strlen(runtime_options.pg_rewind))

+ {

+ maxlen_snprintf(remote_pg_rewind,

+ "%s",

+ runtime_options.pg_rewind);

+ }

+ /* No path supplied - assume in normal bindir */

+ else

+ {

+ maxlen_snprintf(remote_pg_rewind,

+ "%s/pg_rewind",

+ pg_bindir);

+ }

+ }

+ else

+ {

+ use_pg_rewind = false;

+ }

+ }

+

+ /* Sanity checks so we're sure pg_rewind can be used */

+ if (use_pg_rewind == true)

+ {

+ bool wal_log_hints = false;

+

+ /* check pg_rewind actually exists on remote */

+

+ maxlen_snprintf(command,

+ "ls -1 %s >/dev/null 2>&1 && echo 1 || echo 0",

+ remote_pg_rewind);

+

+ initPQExpBuffer(&command_output);

+

+ (void)remote_command(

+ remote_host,

+ runtime_options.remote_user,

+ command,

+ &command_output);

+

+ if (*command_output.data == '0')

+ {

+ log_err(_("unable to find pg_rewind on the remote server\n"));

+ log_err(_("expected location is: %s\n"), remote_pg_rewind);

+ exit(ERR_BAD_CONFIG);

+ }

+

+ /* check that server is appropriately configured */

+

+ /*

+ * "full_page_writes" must be enabled in any case

+ */

+

+ if (guc_set(remote_conn, "full_page_writes", "=", "off"))

+ {

+ log_err(_("\"full_page_writes\" must be set to \"on\""));

+ exit(ERR_BAD_CONFIG);

+ }

+

+ /*

+ * Check whether wal_log_hints is on - if so we're fine and don't need

+ * to check for checksums

+ */

+

+ wal_log_hints = guc_set(remote_conn, "wal_log_hints", "=", "on");

+

+ if (wal_log_hints == false)

+ {

+ char local_data_directory[MAXLEN];

+ int data_checksum_version;

+

+ /*

+ * check the *local* server's control data for the date checksum

+ * version - much easier than doing it on the remote server

+ */

+

+ if (get_pg_setting(local_conn, "data_directory", local_data_directory) == false)

+ {

+ log_err(_("unable to retrieve standby's data directory location\n"));

+ PQfinish(remote_conn);

+ PQfinish(local_conn);

+ exit(ERR_DB_CON);

+ }

+

+ data_checksum_version = get_data_checksum_version(local_data_directory);

+

+ if (data_checksum_version == 0)

+ {

+ log_err(_("pg_rewind cannot be used - data checksums are not enabled for this cluster and \"wal_log_hints\" is \"off\"\n"));

+ exit(ERR_BAD_CONFIG);

+ }

+ }

+ }

+

+ PQfinish(local_conn);

+ PQfinish(remote_conn);

+

+ /* Determine the remote's configuration file location */

+

+ /* Remote configuration file provided - check it exists */

+ if (runtime_options.remote_config_file[0])

+ {

+ log_verbose(LOG_INFO, _("looking for file \"%s\" on remote server \"%s\"\n"),

+ runtime_options.remote_config_file,

+ remote_host);

+

+ maxlen_snprintf(command,

+ "ls -1 %s >/dev/null 2>&1 && echo 1 || echo 0",

+ runtime_options.remote_config_file);

+

+ initPQExpBuffer(&command_output);

+

+ (void)remote_command(

+ remote_host,

+ runtime_options.remote_user,

+ command,

+ &command_output);

+

+ if (*command_output.data == '0')

+ {

+ log_err(_("unable to find the specified repmgr configuration file on remote server\n"));

+ exit(ERR_BAD_CONFIG);

+ }

+

+ termPQExpBuffer(&command_output);

+

+ log_verbose(LOG_INFO, _("remote configuration file \"%s\" found on remote server\n"),

+ runtime_options.remote_config_file);

+

+ termPQExpBuffer(&command_output);

+ }

+ /*

+ * No remote configuration file provided - check some default locations:

+ * - path of configuration file for this repmgr

+ * - /etc/repmgr.conf

+ */

+ else

+ {

+ int i;

+ bool config_file_found = false;

+

+ const char *config_paths[] = {

+ runtime_options.config_file,

+ "/etc/repmgr.conf",

+ NULL

+ };

+

+ log_verbose(LOG_INFO, _("no remote configuration file provided - checking default locations\n"));

+

+ for(i = 0; config_paths[i] && config_file_found == false; ++i)

+ {

+ log_verbose(LOG_INFO, _("checking \"%s\"\n"), config_paths[i]);

+

+ maxlen_snprintf(command,

+ "ls -1 %s >/dev/null 2>&1 && echo 1 || echo 0",

+ config_paths[i]);

+

+ initPQExpBuffer(&command_output);

+

+ (void)remote_command(

+ remote_host,

+ runtime_options.remote_user,

+ command,

+ &command_output);

+

+ if (*command_output.data == '1')

+ {

+ strncpy(runtime_options.remote_config_file, config_paths[i], MAXLEN);

+ log_verbose(LOG_INFO, _("configuration file \"%s\" found on remote server\n"),

+ runtime_options.remote_config_file);

+ config_file_found = true;

+ }

+

+ termPQExpBuffer(&command_output);

+ }

+

+ if (config_file_found == false)

+ {

+ log_err(_("no remote configuration file supplied or found in a default location - terminating\n"));

+ log_hint(_("specify the remote configuration file with -C/--remote-config-file\n"));

+ exit(ERR_BAD_CONFIG);

+ }

+ }

+

+

+

+ /*

+ * Sanity checks completed - prepare for the switchover

+ */

+

+ /*

+ * When using pg_rewind (the preferable option, and default from 9.5

+ * onwards), we need to archive any configuration files in the remote

+ * server's data directory as they'll be overwritten by pg_rewind

+ *

+ * Possible todo item: enable the archive location to be specified

+ * by the user

+ */

+ if (use_pg_rewind == true)

+ {

+ maxlen_snprintf(remote_archive_config_dir,

+ "/tmp/repmgr-%s-archive",

+ remote_node_record.name);

+

+ log_verbose(LOG_DEBUG, "remote_archive_config_dir: %s\n", remote_archive_config_dir);

+

+ maxlen_snprintf(command,

+ "%s/repmgr standby archive-config -f %s --config-archive-dir=%s",

+ pg_bindir,

+ runtime_options.remote_config_file,

+ remote_archive_config_dir);

+

+ log_debug("Executing:\n%s\n", command);

+

+ initPQExpBuffer(&command_output);

+

+ (void)remote_command(

+ remote_host,

+ runtime_options.remote_user,

+ command,

+ &command_output);

+

+ termPQExpBuffer(&command_output);

+ }

+

+ /*

+ * Stop the remote primary

+ *

+ * We'll issue the pg_ctl command but not force it not to wait; we'll check

+ * the connection from here - and error out if no shutdown is detected

+ * after a certain time.

+ *

+ * XXX currently we assume the same Postgres binary path on the primary

+ * as configured on the local standby; we may need to add a command

+ * line option to provide an explicit path (--remote-pg-bindir)?

+ */

+

+ /*

+ * TODO

+ * - notify repmgrd instances that this is a controlled

+ * event so they don't initiate failover

+ * - optional "immediate" shutdown?

+ * -> use -F/--force?

+ */

+

+ maxlen_snprintf(command,

+ "%s/pg_ctl -D %s -m %s -W stop >/dev/null 2>&1 && echo 1 || echo 0",

+ pg_bindir,

+ remote_data_directory,

+ runtime_options.pg_ctl_mode);

+

+ initPQExpBuffer(&command_output);

+

+ // XXX handle failure

+

+ (void)remote_command(

+ remote_host,

+ runtime_options.remote_user,

+ command,

+ &command_output);

+

+ termPQExpBuffer(&command_output);

+

+ connection_success = false;

+

+ /* loop for timeout waiting for current primary to stop */

+

+ for(i = 0; i < options.reconnect_attempts; i++)

+ {

+ /* Check whether primary is available */

+

+ remote_conn = test_db_connection(remote_conninfo, false); /* don't fail on error */

+

+ /* XXX failure to connect doesn't mean the server is necessarily

+ * completely stopped - we need to better detect the reason for

+ * connection failure ("server not listening" vs "shutting down")

+ *

+ * -> check is_pgup()

+ */

+ if (PQstatus(remote_conn) != CONNECTION_OK)

+ {

+ connection_success = true;

+

+ log_notice(_("current master has been stopped\n"));

+ break;

+ }

+ PQfinish(remote_conn);

+

+ // configurable?

+ sleep(options.reconnect_interval);

+ i++;

+ }

+

+ if (connection_success == false)

+ {

+ log_err(_("master server did not shut down\n"));

+ log_hint(_("check the master server status before performing any further actions"));

+ exit(ERR_FAILOVER_FAIL);

+ }

+

+ /* promote this standby */

+

+ do_standby_promote();

+

+ /*

+ * TODO: optionally have any other downstream nodes from old primary

+ * follow new primary? Currently they'll just latch onto the old

+ * primary as cascaded standbys.

+ */

+

+ /* restore old primary */

+

+ /* TODO: additional check old primary is shut down */

+

+ if (use_pg_rewind == true)

+ {

+ PQExpBufferData recovery_done_remove;

+

+ /* Execute pg_rewind */

+ maxlen_snprintf(command,

+ "%s/pg_rewind -D %s --source-server=\\'%s\\'",

+ pg_bindir,

+ remote_data_directory,

+ options.conninfo);

+

+ log_notice("Executing pg_rewind on old master server\n");

+ log_debug("pg_rewind command is:\n%s\n", command);

+

+ initPQExpBuffer(&command_output);

+

+ // XXX handle failure

+

+ (void)remote_command(

+ remote_host,

+ runtime_options.remote_user,

+ command,

+ &command_output);

+

+ termPQExpBuffer(&command_output);

+

+ /* Restore any previously archived config files */

+ maxlen_snprintf(command,

+ "%s/repmgr standby restore-config -D %s --config-archive-dir=%s",

+ pg_bindir,

+ remote_data_directory,

+ remote_archive_config_dir);

+

+ initPQExpBuffer(&command_output);

+

+ // XXX handle failure

+

+ (void)remote_command(

+ remote_host,

+ runtime_options.remote_user,

+ command,

+ &command_output);

+

+ termPQExpBuffer(&command_output);

+

+ /* remove any recovery.done file copied in by pg_rewind */

+

+ initPQExpBuffer(&recovery_done_remove);

+

+ appendPQExpBuffer(&recovery_done_remove,

+ "test -e %s/recovery.done && rm -f %s/recovery.done",

+ remote_data_directory,

+ remote_data_directory);

+ initPQExpBuffer(&command_output);

+

+ // XXX handle failure

+

+ (void)remote_command(

+ remote_host,

+ runtime_options.remote_user,

+ recovery_done_remove.data,

+ &command_output);

+

+ termPQExpBuffer(&command_output);

+ termPQExpBuffer(&recovery_done_remove);

+

+

+

+ }

+ else

+ {

+ /*

+ * For 9.3/9.4, if pg_rewind is not available on the remote server,

+ * we'll need to force a reclone of the standby sing rsync - this may

+ * take some time on larger databases, so use with care!

+ *

+ * Note that following this clone we'll be using `repmgr standby follow`

+ * to start the server - that will mean recovery.conf will be created

+ * for a second time, but as this is a workaround for the absence

+ * of pg_rewind. It's preferable to have `repmgr standby follow` start

+ * the remote database as it can access the remote config file

+ * directly.

+ */

+

+ format_db_cli_params(options.conninfo, repmgr_db_cli_params);

+ maxlen_snprintf(command,

+ "%s/repmgr -D %s -f %s %s --rsync-only --force --ignore-external-config-files standby clone",

+ pg_bindir,

+ remote_data_directory,

+ runtime_options.remote_config_file,

+ repmgr_db_cli_params

+ );

+

+ log_debug("Executing:\n%s\n", command);

+

+ initPQExpBuffer(&command_output);

+

+ (void)remote_command(

+ remote_host,

+ runtime_options.remote_user,

+ command,

+ &command_output);

+

+ termPQExpBuffer(&command_output);

+ }

+

+ /*

+ * Execute `repmgr standby follow` to create recovery.conf and start

+ * the remote server

+ */

+ format_db_cli_params(options.conninfo, repmgr_db_cli_params);

+ maxlen_snprintf(command,

+ "%s/repmgr -D %s -f %s %s standby follow",

+ pg_bindir,

+ remote_data_directory,

+ runtime_options.remote_config_file,

+ repmgr_db_cli_params

+ );

+

+ log_debug("Executing:\n%s\n", command);

+

+ initPQExpBuffer(&command_output);

+

+ (void)remote_command(

+ remote_host,

+ runtime_options.remote_user,

+ command,

+ &command_output);

+

+ termPQExpBuffer(&command_output);

+

+ /* verify that new standby is connected and replicating */

+

+ connection_success = false;

+

+ for(i = 0; i < options.reconnect_attempts; i++)

+ {

+ /* Check whether primary is available */

+

+ remote_conn = test_db_connection(remote_conninfo, false); /* don't fail on error */

+

+ if (PQstatus(remote_conn) == CONNECTION_OK)

+ {

+ log_debug("connected to new standby (old master)\n");

+ if (is_standby(remote_conn) == 0)

+ {

+ log_err(_("new standby (old master) is not a standby\n"));

+ exit(ERR_FAILOVER_FAIL);

+ }

+ connection_success = true;

+ break;

+ }

+ PQfinish(remote_conn);

+

+ sleep(options.reconnect_interval);

+ i++;

+ }

+

+ if (connection_success == false)

+ {

+ log_err(_("unable to connect to new standby (old master)\n"));

+ exit(ERR_FAILOVER_FAIL);

+ }

+

+ log_debug("new standby is in recovery\n");

+

+ /* Check for entry in pg_stat_replication */

+

+ local_conn = establish_db_connection(options.conninfo, true);

+

+

+ query_result = get_node_replication_state(local_conn, remote_node_record.name, remote_node_replication_state);

+ if (query_result == -1)

+ {

+ log_err(_("unable to retrieve replication status for node %i\n"), remote_node_id);

+ PQfinish(local_conn);

+

+ // errcode?

+ exit(ERR_DB_QUERY);

+ }

+

+ if (query_result == 0)

+ {

+ log_err(_("node %i not replicating\n"), remote_node_id);

+ }

+ else

+ {

+ /* XXX other valid values? */

+ /* XXX we should poll for a while in case the node takes time to connect to the primary */

+ if (strcmp(remote_node_replication_state, "streaming") == 0 ||

+ strcmp(remote_node_replication_state, "catchup") == 0)

+ {

+ log_verbose(LOG_NOTICE, _("node %i is replicating in state \"%s\"\n"), remote_node_id, remote_node_replication_state);

+ }

+ else

+ {

+ log_err(_("node %i replication state is \"%s\"\n"), remote_node_id, remote_node_replication_state);

+ PQfinish(local_conn);

+ exit(ERR_DB_QUERY);

+ }

+ }

+

+ /*

+ * If replication slots are in use, and an inactive one for this node

+ * (a former standby) exists on the remote node (a former primary),

+ * drop it.

+ */

+

+ if (options.use_replication_slots)

+ {

+ t_node_info local_node_record;

+

+ query_result = get_node_record(local_conn, options.cluster_name, options.node, &local_node_record);

+

+ remote_conn = establish_db_connection(remote_conninfo, false);

+

+ if (PQstatus(remote_conn) != CONNECTION_OK)

+ {

+ log_warning(_("unable to connect to former master to clean up replication slots \n"));

+ }

+ else

+ {

+ t_replication_slot slot_info;

+ int query_res;

+

+ query_res = get_slot_record(remote_conn, local_node_record.slot_name, &slot_info);

+

+ if (query_res)

+ {

+ if (slot_info.active == false)

+ {

+ if (drop_replication_slot(remote_conn, local_node_record.slot_name) == true)

+ {

+ log_notice(_("replication slot \"%s\" deleted on former master\n"), local_node_record.slot_name);

+ }

+ else

+ {

+ log_err(_("unable to delete replication slot \"%s\" on former master\n"), local_node_record.slot_name);

+ }

+ }

+ /* if active replication slot exists, call Houston as we have a problem */

+ else

+ {

+ log_err(_("replication slot \"%s\" is still active on former master\n"), local_node_record.slot_name);

+ }

+ }

+ }

+

+ PQfinish(remote_conn);

+ }

+

+ /* TODO: verify this node's record was updated correctly */

+

+ PQfinish(local_conn);

+

+ log_notice(_("switchover was successful\n"));

+ return;

+}

+

+

+/*

+ * Intended mainly for "internal" use by `standby switchover`, which

+ * calls this on the target server to archive any configuration files

+ * in the data directory, which may be overwritten by an operation

+ * like pg_rewind

+ */

+static void

+do_standby_archive_config(void)

+{

+ PGconn *local_conn = NULL;

+ char sqlquery[QUERY_STR_LEN];

+ PGresult *res;

+ int i, copied_count = 0;

+

+ if (mkdir(runtime_options.config_archive_dir, S_IRWXU) != 0 && errno != EEXIST)

+ {

+ log_err(_("unable to create temporary directory\n"));

+ exit(ERR_BAD_CONFIG);

+ }

+

+ // XXX check if directory is directory and we own it

+ // XXX delete any files in dir in case it existed already

+

+ local_conn = establish_db_connection(options.conninfo, true);

+

+ /*

+ * Detect which config files are actually inside the data directory;

+ * this query will include any settings from included files too

+ */

+ sqlquery_snprintf(sqlquery,

+ "WITH files AS ( "

+ " WITH dd AS ( "

+ " SELECT setting "

+ " FROM pg_settings "

+ " WHERE name = 'data_directory') "

+ " SELECT distinct(sourcefile) AS config_file"

+ " FROM dd, pg_settings ps "

+ " WHERE ps.sourcefile IS NOT NULL "

+ " AND ps.sourcefile ~ ('^' || dd.setting) "

+ " UNION "

+ " SELECT ps.setting AS config_file"

+ " FROM dd, pg_settings ps "

+ " WHERE ps.name IN ( 'config_file', 'hba_file', 'ident_file') "

+ " AND ps.setting ~ ('^' || dd.setting) "

+ ") "

+ " SELECT config_file, "

+ " regexp_replace(config_file, '^.*\\/','') AS filename "

+ " FROM files "

+ "ORDER BY config_file");

+

+ log_verbose(LOG_DEBUG, "do_standby_archive_config(): %s\n", sqlquery);

+

+ res = PQexec(local_conn, sqlquery);

+ if (PQresultStatus(res) != PGRES_TUPLES_OK)

+ {

+ log_err(_("unable to query config file locations\n"));

+ PQclear(res);

+ PQfinish(local_conn);

+ exit(ERR_DB_QUERY);

+ }

+

+ /* Copy any configuration files to the specified directory */

+ for (i = 0; i < PQntuples(res); i++)

+ {

+ PQExpBufferData buf;

+

+ initPQExpBuffer(&buf);

+ appendPQExpBuffer(&buf, "%s/%s",

+ runtime_options.config_archive_dir, PQgetvalue(res, i, 1));

+

+ log_verbose(LOG_DEBUG, "Copying %s to %s/\n", PQgetvalue(res, i, 0), buf.data);

+ /* XXX check result */

+ copy_file(PQgetvalue(res, i, 0), buf.data);

+

+ termPQExpBuffer(&buf);

+

+ copied_count++;

+ }

+

+ PQclear(res);

+

+ PQfinish(local_conn);

+

+ log_notice(_("%i files copied to %s\n"), copied_count, runtime_options.config_archive_dir);

+}

+

+/*

+ * Intended mainly for "internal" use by `standby switchover`, which

+ * calls this on the target server to restore any configuration files

+ * to the data directory, which may have been overwritten by an operation

+ * like pg_rewind

+ *

+ * Not designed to be called if the instance is running, but does

+ * not currently check.

+ *

+ * Requires -D/--data-dir and --config_archive_dir

+ *

+ * Removes --config_archive_dir after successful copy

+ */

+static void

+do_standby_restore_config(void)

+{

+ DIR *arcdir;

+ struct dirent *arcdir_ent;

+ int copied_count = 0;

+ bool copy_ok = true;

+

+ arcdir = opendir(runtime_options.config_archive_dir);

+ if (arcdir == NULL)

+ {

+ log_err(_("Unable to open directory '%s'\n"), runtime_options.config_archive_dir);

+ exit(ERR_BAD_CONFIG);

+ }

+

+ while ((arcdir_ent = readdir(arcdir)) != NULL) {

+ PQExpBufferData src_file;

+ PQExpBufferData dst_file;

+

+ if (arcdir_ent->d_type != DT_REG)

+ {

+ continue;

+ }

+ initPQExpBuffer(&src_file);

+ initPQExpBuffer(&dst_file);

+

+ appendPQExpBuffer(&src_file, "%s/%s",

+ runtime_options.config_archive_dir, arcdir_ent->d_name);

+

+ appendPQExpBuffer(&dst_file, "%s/%s",

+ runtime_options.dest_dir, arcdir_ent->d_name);

+

+ log_verbose(LOG_DEBUG, "Copying %s to %s\n", src_file.data, dst_file.data);

+

+ /* XXX check result */

+

+ if (copy_file(src_file.data, dst_file.data) == false)

+ {

+ copy_ok = false;

+ log_warning(_("Unable to copy %s from %s\n"), arcdir_ent->d_name, runtime_options.config_archive_dir);

+ }

+ else

+ {

+ unlink(src_file.data);

+ copied_count++;

+ }

+

+ termPQExpBuffer(&src_file);

+ termPQExpBuffer(&dst_file);

+ }

+

+ closedir(arcdir);

+

+

+ if (copy_ok == false)

+ {

+ log_err(_("Unable to copy all files from %s\n"), runtime_options.config_archive_dir);

+ exit(ERR_BAD_CONFIG);

+ }

+

+ log_notice(_("%i files copied to %s\n"), copied_count, runtime_options.dest_dir);

+

+ /*

+ * Finally, delete directory - it should be empty unless it's been interfered

+ * with for some reason, in which case manual attention is required

+ */

+

+ if (rmdir(runtime_options.config_archive_dir) != 0 && errno != EEXIST)

+ {

+ log_err(_("Unable to delete %s\n"), runtime_options.config_archive_dir);

+ exit(ERR_BAD_CONFIG);

+ }

+

+ log_verbose(LOG_NOTICE, "Directory %s deleted\n", runtime_options.config_archive_dir);

+

+ return;

+}

+

+

+ char repmgr_user[MAXLEN];

+ char repmgr_db[MAXLEN];

+ /*

+ * Extract the repmgr user and database names from the conninfo string

+ * provided in repmgr.conf

+ */

+ get_conninfo_value(options.conninfo, "user", repmgr_user);

+ get_conninfo_value(options.conninfo, "dbname", repmgr_db);

+

+ maxlen_snprintf(script, "%s %s -D %s init -o \"%s-U %s\"",

+ make_pg_path("pg_ctl"),

+ options.pg_ctl_options, runtime_options.dest_dir,

+ runtime_options.witness_pwprompt ? "-W " : "",

+ runtime_options.superuser);

+ /*

+ * Attempt to extract a port number from the provided conninfo string.

+ get_conninfo_value(options.conninfo, "port", runtime_options.localport);

+ maxlen_snprintf(script, "%s %s -w -D %s start",

+ make_pg_path("pg_ctl"),

+ options.pg_ctl_options, runtime_options.dest_dir);

+

+ if (strcmp(repmgr_user, "postgres") != 0)

+ {

+ /* create required user; needs to be superuser to create untrusted

+ * language function in C */

+ maxlen_snprintf(script, "%s -p %s --superuser --login %s-U %s %s",

+ make_pg_path("createuser"),

+ runtime_options.localport,

+ runtime_options.witness_pwprompt ? "-P " : "",

+ runtime_options.superuser,

+ repmgr_user);

+ maxlen_snprintf(script, "%s -p %s -U %s --owner=%s %s",

+ make_pg_path("createdb"),

+ runtime_options.localport,

+ runtime_options.superuser,

+ repmgr_user,

+ repmgr_db);

+ char *errmsg = _("Unable to retrieve location of pg_hba.conf");

+ char *errmsg = _("Unable to copy pg_hba.conf from master");

+ /* reload witness server to activate the copied pg_hba.conf */

+ maxlen_snprintf(script, "%s %s -w -D %s reload",

+ make_pg_path("pg_ctl"),

+ options.pg_ctl_options, runtime_options.dest_dir);

+ /* establish a connection to the witness, and create the schema */

+ witnessconn = establish_db_connection(options.conninfo, false);

+

+ if (PQstatus(witnessconn) != CONNECTION_OK)

+ {

+ create_event_record(masterconn,

+ &options,

+ options.node,

+ "witness_create",

+ false,

+ _("Unable to connect to witness servetr"));

+ PQfinish(masterconn);

+ exit(ERR_BAD_CONFIG);

+ }

+

+

+ log_info(_("starting copy of configuration from master...\n"));

+

+ begin_transaction(witnessconn);

+

+ if (!create_schema(witnessconn))

+ {

+ rollback_transaction(witnessconn);

+ create_event_record(masterconn,

+ &options,

+ options.node,

+ "witness_create",

+ false,

+ _("Unable to create schema on witness"));

+ PQfinish(masterconn);

+ PQfinish(witnessconn);

+ exit(ERR_BAD_CONFIG);

+ }

+

+ commit_transaction(witnessconn);

+

+ /*

+ * Register new witness server on the primary

+ * Do this as late as possible to avoid having to delete

+ * the record if the server creation fails

+ */

+

+ /*

+ * delete previously created witness node record

+ * XXX maybe set inactive?

+ */

+ delete_node_record(masterconn,

+ options.node,

+ "witness create");

+

+ if (strcmp(repmgr_user, "postgres") != 0)

+ sqlquery_snprintf(sqlquery, "ALTER ROLE %s NOSUPERUSER", repmgr_user);

+ repmgr_user, sqlquery);

+ log_err(_("Unable to alter user privileges for user %s: %s\n"),

+ repmgr_user,

+ /* Finished with the witness server */

+

+ PQfinish(witnessconn);

+

+do_help(void)

+ const char *host;

+

+ printf(_(" %s [OPTIONS] standby {register|unregister|clone|promote|follow|switchover}\n"),

+ printf(_(" -L, --log-level set log level (overrides configuration file; default: NOTICE)\n"));

+ printf(_(" -d, --dbname=DBNAME database to connect to (default: \"%s\")\n"), runtime_options.dbname);

+ host = getenv("PGHOST");

+ printf(_(" -h, --host=HOSTNAME database server host or socket directory (default: \"%s\")\n"), host ? host : _("local socket"));

+ printf(_(" -p, --port=PORT database server port (default: \"%s\")\n"), runtime_options.masterport);

+ printf(_(" -U, --username=USERNAME database user name to connect as (default: \"%s\")\n"), runtime_options.username);

+ printf(_(" -m, --mode (standby switchover) shutdown mode (smart|fast|immediate)\n"));

+ printf(_(" -C, --remote-config-file (standby switchover) path to the configuration file on\n" \

+ " the current master\n"));

+ printf(_(" --pg_rewind[=VALUE] (standby switchover) 9.3/9.4 only - use pg_rewind if available,\n" \

+ " optionally providing a path to the binary\n"));

+ printf(_(" -k, --keep-history=VALUE (cluster cleanup) retain indicated number of days of history (default: 0)\n"));

+/* printf(_(" --initdb-no-pwprompt (witness server) no superuser password prompt during initdb\n"));*/

+ printf(_(" -P, --pwprompt (witness server) prompt for password when creating users\n"));

+ printf(_(" master register - registers the master in a cluster\n"));

+ printf(_(" standby clone [node] - creates a new standby\n"));

+ printf(_(" standby register - registers a standby in a cluster\n"));

+ printf(_(" standby unregister - unregisters a standby in a cluster\n"));

+ printf(_(" standby promote - promotes a specific standby to master\n"));

+ printf(_(" standby follow - makes standby follow a new master\n"));

+ printf(_(" standby switchover - switch this standby with the current master\n"));

+ printf(_(" witness create - creates a new witness server\n"));

+ printf(_(" cluster show - displays information about cluster nodes\n"));

+ printf(_(" cluster cleanup - prunes or truncates monitoring history\n" \

+ " (monitoring history creation requires repmgrd\n" \

+ " with --monitoring-history option)\n"));

+ maxlen_snprintf(script, "ssh -o Batchmode=yes %s %s %s 2>/dev/null",

+ maxlen_snprintf(script, "ssh -o Batchmode=yes %s %s -l %s %s 2>/dev/null",

+ log_verbose(LOG_DEBUG, _("test_ssh_connection(): executing %s\n"), script);

+ appendPQExpBuffer(&rsync_flags, "%s",

+ " --exclude=recovery.conf --exclude=recovery.done");

+

+ if (connection_param_provided)

+ if (connection_param_provided)

+ if (connection_param_provided)

+ if (connection_param_provided)

+

+ * we will try to detect the master in repl_nodes; if we can't find

+

+ if (runtime_options.host[0] || runtime_options.dest_dir[0])

+ if (!runtime_options.host[0])

+ {

+ error_list_append(&cli_errors, _("master hostname (-h/--host) required when executing STANDBY FOLLOW with -D/--data-dir option"));

+ }

+

+ if (!runtime_options.dest_dir[0])

+ {

+ error_list_append(&cli_errors, _("local data directory (-D/--data-dir) required when executing STANDBY FOLLOW with -h/--host option"));

+ }

+

+ if (runtime_options.fast_checkpoint && runtime_options.rsync_only)

+ {

+ error_list_append(&cli_warnings, _("-c/--fast-checkpoint has no effect when using -r/--rsync-only"));

+ }

+ config_file_required = false;

+ break;

+ case STANDBY_SWITCHOVER:

+ /* allow all parameters to be supplied */

+ break;

+ case STANDBY_ARCHIVE_CONFIG:

+ if (strcmp(runtime_options.config_archive_dir, "") == 0)

+ error_list_append(&cli_errors, _("--config-archive-dir required when executing STANDBY ARCHIVE_CONFIG"));

+ }

+ break;

+ case STANDBY_RESTORE_CONFIG:

+ if (strcmp(runtime_options.config_archive_dir, "") == 0)

+ {

+ error_list_append(&cli_errors, _("--config-archive-dir required when executing STANDBY RESTORE_CONFIG"));

+ if (strcmp(runtime_options.dest_dir, "") == 0)

+ error_list_append(&cli_errors, _("-D/--data-dir required when executing STANDBY RESTORE_CONFIG"));

+ /* Require data directory */

+ if (strcmp(runtime_options.dest_dir, "") == 0)

+ {

+ error_list_append(&cli_errors, _("-D/--data-dir required when executing WITNESS CREATE"));

+ }

+ /* Warn about parameters which apply to STANDBY SWITCHOVER only */

+ if (action != STANDBY_SWITCHOVER)

+ {

+ if (pg_rewind_supplied == true)

+ {

+ error_list_append(&cli_warnings, _("--pg_rewind can only be used when executing STANDBY SWITCHOVER"));

+ }

+ }

+

+

+

+ /* CREATE VIEW repl_show_nodes */

+ sqlquery_snprintf(sqlquery,

+ "CREATE VIEW %s.repl_show_nodes AS "

+ "SELECT rn.id, rn.conninfo, rn.type, rn.name, rn.cluster,"

+ " rn.priority, rn.active, sq.name AS upstream_node_name"

+ " FROM %s.repl_nodes as rn"

+ " LEFT JOIN %s.repl_nodes AS sq"

+ " ON sq.id=rn.upstream_node_id",

+ get_repmgr_schema_quoted(conn),

+ get_repmgr_schema_quoted(conn),

+ get_repmgr_schema_quoted(conn));

+

+ log_debug(_("master register: %s\n"), sqlquery);

+

+ res = PQexec(conn, sqlquery);

+ if (!res || PQresultStatus(res) != PGRES_COMMAND_OK)

+ {

+ log_err(_("unable to create view %s.repl_show_nodes: %s\n"),

+ get_repmgr_schema_quoted(conn), PQerrorMessage(conn));

+

+ if (res != NULL)

+ PQclear(res);

+

+ return false;

+ }

+ PQclear(res);

+

+

+ * If archive_mode is enabled, check that 'archive_command' is non empty

+ * (however it's not practical to check that it actually represents a valid

+ * command).

+ * From PostgreSQL 9.5, archive_mode can be one of 'off', 'on' or 'always'

+ * so for ease of backwards compatibility, rather than explicitly check for an

+ * enabled mode, check that it's not "off".

+

+ if (guc_set(conn, "archive_mode", "!=", "off"))

+ * for the primary server, however the assumption is that we'll be

+ * cloning standbys and thus copying the primary configuration;

+ * this way the standby will be correctly configured by default.

+

+

+

+/*

+ * Execute a command via ssh on the remote host.

+ *

+ * TODO: implement SSH calls using libssh2.

+ */

+static bool

+remote_command(const char *host, const char *user, const char *command, PQExpBufferData *outputbuf)

+{

+ FILE *fp;

+ char ssh_command[MAXLEN];

+ PQExpBufferData ssh_host;

+

+ char output[MAXLEN];

+

+ initPQExpBuffer(&ssh_host);

+

+ if (*user != '\0')

+ {

+ appendPQExpBuffer(&ssh_host, "%s@", user);

+ }

+

+ appendPQExpBuffer(&ssh_host, "%s",host);

+

+ maxlen_snprintf(ssh_command,

+ "ssh -o Batchmode=yes %s %s",

+ ssh_host.data,

+ command);

+

+ termPQExpBuffer(&ssh_host);

+

+ log_debug("remote_command(): %s\n", ssh_command);

+

+ fp = popen(ssh_command, "r");

+

+ if (fp == NULL)

+ {

+ log_err(_("unable to execute remote command:\n%s\n"), ssh_command);

+ return false;

+ }

+

+ /* TODO: better error handling */

+ while (fgets(output, MAXLEN, fp) != NULL)

+ {

+ appendPQExpBuffer(outputbuf, "%s", output);

+ }

+

+ pclose(fp);

+

+ log_verbose(LOG_DEBUG, "remote_command(): output returned was:\n%s", outputbuf->data);

+

+ return true;

+}

+

+

+/*

+ * Extract values from provided conninfo string and return

+ * formatted as command-line parameters suitable for passing to repmgr

+ */

+static void

+format_db_cli_params(const char *conninfo, char *output)

+{

+ PQExpBufferData buf;

+ char host[MAXLEN] = "";

+ char port[MAXLEN] = "";

+ char dbname[MAXLEN] = "";

+ char user[MAXLEN] = "";

+

+ initPQExpBuffer(&buf);

+

+ get_conninfo_value(conninfo, "host", host);

+ get_conninfo_value(conninfo, "port", port);

+ get_conninfo_value(conninfo, "dbname", dbname);

+ get_conninfo_value(conninfo, "user", user);

+

+ if (host[0])

+ {

+ appendPQExpBuffer(&buf, "-h %s ", host);

+ }

+

+ if (port[0])

+ {

+ appendPQExpBuffer(&buf, "-p %s ", port);

+ }

+

+ if (dbname[0])

+ {

+ appendPQExpBuffer(&buf, "-d %s ", dbname);

+ }

+

+ if (user[0])

+ {

+ appendPQExpBuffer(&buf, "-U %s ", user);

+ }

+

+ strncpy(output, buf.data, MAXLEN);

+

+ termPQExpBuffer(&buf);

+

+}

+

+bool

+copy_file(const char *old_filename, const char *new_filename)

+{

+ FILE *ptr_old, *ptr_new;

+ int a;

+

+ ptr_old = fopen(old_filename, "r");

+ ptr_new = fopen(new_filename, "w");

+

+ if (ptr_old == NULL)

+ return false;

+

+ if (ptr_new == NULL)

+ {

+ fclose(ptr_old);

+ return false;

+ }

+

+ chmod(new_filename, S_IRUSR | S_IWUSR);

+

+ while(1)

+ {

+ a = fgetc(ptr_old);

+

+ if (!feof(ptr_old))

+ {

+ fputc(a, ptr_new);

+ }

+ else

+ {

+ break;

+ }

+ }

+

+ fclose(ptr_new);

+ fclose(ptr_old);

+

+ return true;

+}

+

+# Some configuration items will be set with a default value; this

+# is noted for each item. Where no default value is shown, the

+# parameter will be treated as empty or false.

+

+upstream_node=1

+# use physical replication slots - PostgreSQL 9.4 and later only

+use_replication_slots=0

+

+logfile='/var/log/repmgr/repmgr.log'

+event_notification_command='/path/to/some/script %n %e %s "%t" "%d"'

+event_notifications=master_register,standby_register,witness_create

+pg_bindir=/usr/bin/

+rsync_options=--archive --checksum --compress --progress --rsh="ssh -o \"StrictHostKeyChecking no\""

+ssh_options=-o "StrictHostKeyChecking no"

+# external command arguments. Values shown are examples.

+pg_ctl_options='-s'

+pg_basebackup_options='--xlog-method=s'

+# These settings are only applied when repmgrd is running. Values shown

+# are defaults.

+# deciding it has failed.

+# Number of attempts at what interval (in seconds) to try and

+# connect to a server to establish its status (e.g. master

+# during failover)

+failover=manual # one of 'automatic', 'manual'

+ # (default: manual)

+ # (default: 100)

+monitor_interval_secs=2

+retry_promote_interval_secs=300

+ * Copyright (c) 2ndQuadrant, 2010-2016

+ bool witness_pwprompt;

+ char pg_ctl_mode[MAXLEN];

+ /*

+ * configuration file parameters which can be overridden on the

+ * command line

+ */

+ /* parameter used by STANDBY SWITCHOVER */

+ char remote_config_file[MAXLEN];

+ char pg_rewind[MAXFILENAME];

+ /* parameter used by STANDBY {ARCHIVE_CONFIG | RESTORE_CONFIG} */

+ char config_archive_dir[MAXLEN];

+

+ /* deprecated command line options */

+ char localport[MAXLEN];

+ bool initdb_no_pwprompt;

+#define T_RUNTIME_OPTIONS_INITIALIZER { "", "", "", "", "", "", "", DEFAULT_WAL_KEEP_SEGMENTS, false, false, false, false, false, false, false, false, false, "smart", "", "", "", "", "", 0, "", "", "", false }

+extern bool config_file_found;

+ * Copyright (C) 2ndQuadrant, 2010-2016

+

+/*

+ * This view shows the list of nodes with the information of which one is the upstream

+ * in each case (when appliable)

+ */

+CREATE VIEW repl_show_nodes AS

+SELECT rn.id, rn.conninfo, rn.type, rn.name, rn.cluster,

+ rn.priority, rn.active, sq.name AS upstream_node_name

+FROM repl_nodes as rn LEFT JOIN repl_nodes AS sq ON sq.id=rn.upstream_node_id;

+ * Copyright (C) 2ndQuadrant, 2010-2016

+ /* Disallow running as root to prevent directory ownership problems */

+ if (geteuid() == 0)

+ {

+ fprintf(stderr,

+ _("%s: cannot be run as root\n"

+ "Please log in (using, e.g., \"su\") as the "

+ "(unprivileged) user that owns "

+ "the data directory.\n"

+ ),

+ progname());

+ exit(1);

+ }

+

+

+ log_hint(

+ log_notice(_("node %d is the best candidate for new master, attempting to follow...\n"),

+ _("node %i now following new upstream node %i"),

+ log_notice("%s\n", event_details.data);

+ _("node %i is now following upstream node %i"),

+ int res;

+ res = get_node_record(conn, cluster, node_id, &node_info);

+ if (res == -1)

+ PQfinish(conn);

+ if (res == 0)

+ {

+# Copyright (c) 2ndQuadrant, 2010-2016

+/*

+ * Update a repmgr 3.0 installation to repmgr 3.1

+ * ----------------------------------------------

+ *

+ * The new repmgr package should be installed first. Then

+ * carry out these steps:

+ *

+ * 1. (If repmgrd is used) stop any running repmgrd instances

+ * 2. On the master node, execute the SQL statements listed below

+ * 3. (If repmgrd is used) restart repmgrd

+ */

+

+/*

+ * If your repmgr installation is not included in your repmgr

+ * user's search path, please set the search path to the name

+ * of the repmgr schema to ensure objects are installed in

+ * the correct location.

+ *

+ * The repmgr schema is "repmgr_" + the cluster name defined in

+ * 'repmgr.conf'.

+ */

+

+-- SET search_path TO 'name_of_repmgr_schema';

+

+BEGIN;

+

+-- New view "repl_show_nodes" which also displays the server's

+-- upstream node

+

+CREATE VIEW repl_show_nodes AS

+SELECT rn.id, rn.conninfo, rn.type, rn.name, rn.cluster,

+ rn.priority, rn.active, sq.name AS upstream_node_name

+FROM repl_nodes as rn LEFT JOIN repl_nodes AS sq ON sq.id=rn.upstream_node_id;

+

+COMMIT;

+ * Copyright (c) 2ndQuadrant, 2010-2016

+ * Copyright (c) 2ndQuadrant, 2010-2016

+ * Copyright (C) 2ndQuadrant, 2010-2016

+ * Copyright (C) 2ndQuadrant, 2010-2016

+ * Copyright (C) 2ndQuadrant, 2010-2016

+#define REPMGR_VERSION "3.1.1"