Estimated Reading time: 6 minutes | Last modified: 2024-11-04, 10:25:08 am

High-Availability Database Cluster Management Tool for OpenResty Edge

Introduction

OpenResty Edge DB Cluster Management Tool is a command-line utility for managing Database high-availability clusters. It helps users quickly set up and manage a Database cluster with monitor and data nodes.

Architecture

The cluster consists of:

  • One monitor node: Manages the cluster state and coordinates failover
  • Two or more data nodes: One primary and one or more standby nodes

System Requirements

  • Operating System: Linux (with systemd support)
  • Memory: Minimum 2GB
  • Disk Space: Minimum 5GB, should be sized according to actual database storage requirements
  • OpenResty PostgreSQL 12 or higher version installed
  • Firewall ports: 5432 (PostgreSQL)
  • Network: All nodes must be able to communicate with each other

Features

  • Install, start, and stop monitor and data nodes
  • Query database cluster status
  • Manage Database HBA(Host-Based Authentication) configuration
  • Support cluster switchover operations
  • Automatic failover capability

Important Notes

  1. Test in a non-production environment first
  2. Always backup important data before installation, you can refer to this document
  3. Perform operations during maintenance windows
  4. When switching monitor nodes, perform switchover on the primary data node first (if alive), then perform switchover on the secondary data nodes
  5. Use the -y option with caution as it skips all confirmations and may lead to data inconsistency, corruption, or loss.

Installation Steps

  1. Download the management tool:
curl -O https://openresty.com/client/oredge/openresty-edge-db-cluster-manager.sh
  1. Install monitor node:
sudo bash openresty-edge-db-cluster-manager.sh -a install -t monitor -n MONITOR_IP
  1. Install one or more data nodes: (The first successfully installed node will become the primary node)
sudo bash openresty-edge-db-cluster-manager.sh -a install -t data -m MONITOR_IP -n NODE_IP
  1. Check the cluster state:
sudo bash openresty-edge-db-cluster-manager.sh -a get -c "cluster state" -m MONITOR_IP

Command Usage

Basic Syntax

sudo bash openresty-edge-db-cluster-manager.sh [options]

Command Options

-a, --action       Action (install/start/stop/get/set/switchover/uninstall)
-t, --type         Node type (monitor/data)
-m, --monitor-ip   Monitor node IP address
-n, --node-ip      Current node IP address
-v, --pg-version   PostgreSQL version (default: 12)
-y, --yes          Skip all confirmations and use default values
                   (Warning: This will auto-confirm destructive operations)
-c, --config       Configuration type (get: cluster state/node type/service status, set: pghba)
-i, --pg-hba-ips   PostgreSQL hba IPs (comma separated)
-s, --switch-type  Switchover type (switch to new monitor)
-h, --help         Show help message

Common Usage Examples

Run without any options to perform all operations interactively:

sudo bash openresty-edge-db-cluster-manager.sh

Install monitor node:

sudo bash openresty-edge-db-cluster-manager.sh -a install -t monitor -n 192.168.1.10

Install data node:

sudo bash openresty-edge-db-cluster-manager.sh -a install -t data -m 192.168.1.10 -n 192.168.1.11

Check cluster status:

sudo bash openresty-edge-db-cluster-manager.sh -a get -c "cluster state" -m 192.168.1.10

Get node type:

sudo bash openresty-edge-db-cluster-manager.sh -a get -c "node type"

Configure PostgreSQL HBA:

sudo bash openresty-edge-db-cluster-manager.sh -a set -c pghba -t data -i "192.168.1.10,192.168.1.11"

Switch to new monitor:

sudo bash openresty-edge-db-cluster-manager.sh -a switchover -s "switch to new monitor" -m 192.168.1.12

Uninstall node:

sudo bash openresty-edge-db-cluster-manager.sh -a uninstall -t monitor

Monitor Node Switchover Scenarios

This section describes detailed steps for monitor node switchover in different scenarios. The initial cluster consists of: 1 monitor node, 1 primary data node, and 1 secondary data node.

Important Notes:

  1. Ensure the new monitor node is properly installed and running before performing switchover
  2. Database service may be briefly affected during switchover
  3. Verify cluster state after each switchover
  4. Perform switchover during maintenance windows
  5. Backup important data before operation
  6. Keep detailed operation logs for troubleshooting
  7. When the monitor node is functioning normally, switchover is usually unnecessary. If you wish to perform a switchover, please stop the monitor node first.

You can use the following command on data nodes to get the node type:

sudo bash openresty-edge-db-cluster-manager.sh -a get -c "node type"
  • monitor: Monitor node
  • primary: Primary data node
  • secondary: Standby data node

Scenario 1: Recoverable Monitor Node

When the monitor node experiences temporary failure but is recoverable, there is no need to perform monitor node switchover:

  1. Fix the monitor node issue

  2. Restart the monitor node service:

    sudo bash openresty-edge-db-cluster-manager.sh -a start -t monitor

  3. Verify cluster state:

    sudo bash openresty-edge-db-cluster-manager.sh -a get -c "cluster state" -m MONITOR_IP

Scenario 2: Unrecoverable Monitor Node with All Data Nodes Normal

When the monitor node cannot be recovered, but all data nodes are running normally, migrate all data nodes to a new monitor node.

  1. Install new monitor node on a new server:

    sudo bash openresty-edge-db-cluster-manager.sh -a install -t monitor -n NEW_MONITOR_IP

  2. Switch primary data node to new monitor node:

    sudo bash openresty-edge-db-cluster-manager.sh -a switchover -s "switch to new monitor" -m NEW_MONITOR_IP

  3. Perform the same switchover operation on secondary data node:

    sudo bash openresty-edge-db-cluster-manager.sh -a switchover -s "switch to new monitor" -m NEW_MONITOR_IP

  4. Verify cluster state:

    sudo bash openresty-edge-db-cluster-manager.sh -a get -c "cluster state" -m NEW_MONITOR_IP

Scenario 3: Monitor and Secondary Node Failure, Primary Node Normal

When both monitor node and secondary data node fail, but primary data node is running normally:

  1. Install new monitor node on a new server:

    sudo bash openresty-edge-db-cluster-manager.sh -a install -t monitor -n NEW_MONITOR_IP

  2. Switch primary data node to new monitor node:

    sudo bash openresty-edge-db-cluster-manager.sh -a switchover -s "switch to new monitor" -m NEW_MONITOR_IP

  3. Verify cluster state:

    sudo bash openresty-edge-db-cluster-manager.sh -a get -c "cluster state" -m NEW_MONITOR_IP

  4. Fix or reinstall secondary node (optional)

Scenario 4: Monitor and Primary Node Failure, Secondary Node Normal

When both monitor node and primary data node fail, but secondary data node is running normally:

  1. Install new monitor node on a new server:

    sudo bash openresty-edge-db-cluster-manager.sh -a install -t monitor -n NEW_MONITOR_IP

  2. Switch secondary data node to new monitor node:

    sudo bash openresty-edge-db-cluster-manager.sh -a switchover -s "switch to new monitor" -m NEW_MONITOR_IP

    When prompted “Will promote current node to primary. Continue?”, select “y” to promote this node to primary.

  3. Verify cluster state:

    sudo bash openresty-edge-db-cluster-manager.sh -a get -c "cluster state" -m NEW_MONITOR_IP

  4. Fix or reinstall original primary node as secondary node (optional)

Logs and History

  • Each operation creates a separate history directory: oredge-db-cluster-histories/<timestamp>
  • All operation logs are saved in oredge-db-cluster-histories/<timestamp>/script.log

Error Handling

If you encounter errors:

  1. Check the log file for detailed information
  2. Verify network connectivity between nodes
  3. Confirm that system requirements are met
  4. Ensure there are no port conflicts
  5. Verify database service installation

FAQ

  1. What happens during a monitor node failure?

    • The data nodes continue to operate
    • You can switch to a new monitor node using the switchover command

  2. How is automatic failover triggered?

    • The monitor node continuously checks data node health
    • If the primary node fails, a standby is automatically promoted

  3. Can I add more nodes to an existing cluster?

    • Yes, use the install command with data node type
    • Existing data on the new node will be cleared and automatically synchronized with the primary node

  4. What should I do if a node fails to start?