How to Build a High-Availability (HA) Cluster on Bare Metal with HAProxy, Keepalived & MariaDB Galera

Phase 1 Architecture Explanation (Visualizing the Setup)

Before touching the command line, you need a clear mental model of the topology. We are distributing our services across three isolated tiers.

IP Addressing Scheme

Phase 2 OS & Network Preparation

Security and kernel tuning are critical.

1. Secure Internal Communication (Firewall)

On all Web and DB nodes, configure Uncomplicated Firewall (ufw) to deny incoming public traffic but explicitly allow required ports from your Private VLAN (10.0.0.0/24):

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh

# Allow internal Web Traffic
sudo ufw allow from 10.0.0.0/24 to any port 80

sudo ufw allow from 10.0.0.0/24 to any port 443

# Allow internal Galera & MySQL Traffic
sudo ufw allow from 10.0.0.0/24 to any port 3306  # MySQL
sudo ufw allow from 10.0.0.0/24 to any port 4444  # Galera SST
sudo ufw allow from 10.0.0.0/24 to any port 4567  # Galera Cluster
sudo ufw allow from 10.0.0.0/24 to any port 4568  # Galera IST

sudo ufw enable
    

2. Sysctl Tuning for High Throughput

To handle high network traffic and allow HAProxy to bind to the Floating IP, apply these kernel parameters. Run this on all 7 nodes:

cat <<EOF | sudo tee /etc/sysctl.d/99-ha-cluster.conf
# Allow HAProxy to bind to the floating IP
net.ipv4.ip_nonlocal_bind = 1

# Increase network connection queues
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
EOF

sudo sysctl --system
    

Phase 3 Setting Up the Database Cluste

We will set up a MariaDB Galera Cluster. It requires at least 3 nodes to maintain quorum.

1. Install MariaDB, Galera, and Backup Tools

Run this on DB-01, DB-02, and DB-03:

sudo apt update
sudo apt install mariadb-server mariadb-client galera-4 mariadb-backup -y
sudo systemctl stop mariadb
    

2. Configure Galera Replication

On all three DB nodes, edit /etc/mysql/mariadb.conf.d/60-galera.cnf.

[galera]
wsrep_on                 = ON
bind-address             = 0.0.0.0
wsrep_provider           = /usr/lib/galera/libgalera_smm.so
wsrep_cluster_name       = "ha_production_cluster"
wsrep_cluster_address    = "gcomm://10.0.0.30,10.0.0.31,10.0.0.32"
binlog_format            = row
default_storage_engine   = InnoDB
innodb_autoinc_lock_mode = 2

# SST Configuration for syncing new nodes
wsrep_sst_method         = mariabackup
# In production, secure this by setting wsrep_sst_auth="sstuser:password"

# Change these per node:
wsrep_node_name          = "DB-01"
wsrep_node_address       = "10.0.0.30"
    

3. Bootstrap the Cluster

On DB-01 ONLY, run:

sudo galera_new_cluster
    

Check if the cluster size is 1:

mysql -u root -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
    

Now, on DB-02 and DB-03, start the service normally:

sudo systemctl start mariadb
sudo systemctl enable mariadb
    

Checking wsrep_cluster_size again on any node should now return 3.

4. Test Replication

On DB-01:

CREATE DATABASE testdb;
CREATE TABLE testdb.users (id INT PRIMARY KEY, name VARCHAR(50));
INSERT INTO testdb.users VALUES (1, 'SysAdmin');
    

On DB-03, verify the data synced instantly:

SELECT * FROM testdb.users;
    

Phase 4 Web Server Configuration

1. Install Nginx

On WEB-01 and WEB-02:

sudo apt install nginx -y
    

2. Create the Health Check Endpoint

HAProxy needs a way to verify if Nginx is healthy. Create a /healthz endpoint on both Web nodes:

echo "OK" | sudo tee /var/www/html/healthz
    

3. Deploying Web Files

In a production environment, application code should normally be deployed through a CI/CD pipeline or a synchronization tool such as lsyncd or rsync. Shared storage such as NFS is generally reserved for user-uploaded content and should itself be deployed in a highly available configuration.

Start Nginx on both nodes:

sudo systemctl enable --now nginx
    

Phase 5 Load Balancing & Auto-Failover

We will configure LB-01 and LB-02 to route traffic and handle Floating IP failover.

1. Install Keepalived and HAProxy

On LB-01 and LB-02:

sudo apt install keepalived haproxy -y
    

2. Configure Keepalived (Unicast VRRP)

Modern bare-metal and cloud environments often block multicast traffic. We will configure Keepalived to use Unicast. Before configuring, run ip addr to find your actual network interface name (e.g., ens18, enp3s0, or eno1).

On LB-01 (MASTER): create /etc/keepalived/keepalived.conf:

vrrp_script chk_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    weight 20
}

vrrp_instance VI_1 {
    state MASTER
    interface ens18              # Replace with your actual network interface name
    virtual_router_id 51
    priority 100
    advert_int 1

    # Unicast configuration
    unicast_src_ip 10.0.0.10     # LB-01 Private IP
    unicast_peer {
        10.0.0.11                # LB-02 Private IP
    }

    authentication {
        auth_type PASS
        auth_pass Secr3tP@ss
    }

    virtual_ipaddress {
        203.0.113.100/32         # The Floating VIP
    }

    track_script {
        chk_haproxy
    }
}
    

On LB-02 (BACKUP): Use the same config, but change:

• state BACKUP
• priority 90
• unicast_src_ip 10.0.0.11
• unicast_peer { 10.0.0.10 }

Restart Keepalived on both nodes:

sudo systemctl enable --now keepalived
    

3. Configure HAProxy

Edit /etc/haproxy/haproxy.cfg on both LB nodes:

frontend public_web_front
    bind 203.0.113.100:80
    mode http
    default_backend internal_web_cluster

backend internal_web_cluster
    mode http
    balance roundrobin
    
    option httpchk GET /healthz
    http-check expect status 200
    
    server web-01 10.0.0.20:80 check inter 2000 fall 3 rise 2
    server web-02 10.0.0.21:80 check inter 2000 fall 3 rise 2

    

Restart HAProxy:

sudo systemctl enable --now haproxy
    

4. Testing the Architecture

• Test Web Failover: Access [http://203.0.113.100/healthz](http://203.0.113.100/healthz) in your browser. Shut down Nginx on WEB-01. HAProxy will detect the failed health check and route all traffic to WEB-02 instantly.
• Test LB Failover: Reboot LB-01 (The MASTER node). Keepalived on LB-02 will immediately detect the missing heartbeat, authenticate, and claim the VIP over unicast.

y deploying this architecture, you have successfully eliminated the primary infrastructure-level single points of failure within your application stack.