Building a Proxmox Cluster with High Availability on Dedicated Servers

Answer first: what it takes, what you get

The requirements, per Proxmox's own documentation and unbent by wishful thinking:

• Three nodes minimum for reliable quorum. (A two-node cluster works with an external QDevice as tie-breaker — details below — but three is where the design wants to be.)
• A reliable, low-latency network between nodes — under 5 ms — with cluster traffic ideally on its own NIC. This is a same-site, private-LAN requirement, not something to stretch across regions.
• Storage the surviving nodes can access: hyperconverged Ceph, per-guest ZFS replication, or external shared storage.

What you get for it, stated equally plainly: when a node dies, HA restarts its guests on surviving nodes, automatically, within a few minutes. That's crash recovery without a human — hugely valuable — but it is not zero-downtime fault tolerance: the guests reboot. Live migration (for planned moves) is the seamless one; failover is a restart.

If your actual need is "recover from failures within a business day", a good backup and restore practice on one solid host is less machinery. Clustering earns its complexity when minutes matter, or when you're running enough guests that single-pane management and live migration pay for themselves daily.

Clustering before HA: worth it on its own

A common conflation: cluster ≠ HA. Joining nodes into a cluster (pvecm) gives you, with no HA configured at all: one web UI managing every node, live migration of VMs between nodes (hardware maintenance without downtime), cluster-wide SDN networking and firewall config, and shared job/permission management. Many production clusters run exactly this — humans decide when guests move — and add HA selectively later, per guest, for the services that justify automated failover.

Quorum: why three nodes

The problem quorum solves is split-brain. If cluster links fail and nodes can't see each other, each side must decide: "am I the cluster now?" If both sides say yes, both start the same VMs on the same shared storage, and you learn what corrupted disk images look like. The rule that prevents this: only a group holding a majority of votes may act. With three nodes, any partition has a majority side (2) and a minority side (1) that freezes; with two nodes, a broken link makes both sides minorities — neither can act, and availability is now worse than a single node.

Hence the two sanctioned shapes: three-plus nodes, or two nodes plus a QDevice — a tiny corosync witness daemon on any third machine (a VPS or small server works; it doesn't run guests, it just votes). What's never sanctioned is the workaround forums suggest at midnight: manually setting expected votes to 1 to force a partitioned node active. That switch is exactly how split-brain becomes a personal experience.

The corosync network: latency is the law

Corosync — the cluster's membership-and-messaging layer — is the component whose requirements are least negotiable and most often violated:

• Latency under 5 ms between all nodes. This is LAN physics: same site, private network. Nodes in different cities aren't a cluster; they're two clusters and a replication strategy.
• Bandwidth is irrelevant; contention is fatal. Corosync needs kilobits — but the moment it shares a link with backup jobs, storage traffic, or VM migration and that link saturates, membership packets arrive late, nodes get declared dead, and HA starts "recovering" guests that were fine. The official recommendation, worth following literally: a dedicated NIC (1 Gbit is plenty) for cluster traffic, and up to eight redundant corosync links if you have spare paths — at minimum, a second link on any other network as fallback.
• Storage traffic (Ceph especially) likewise wants its own lane — see below.

On our dedicated servers this maps cleanly: the private network between your machines carries corosync (and storage) on interfaces that never contend with public traffic.

Storage: the real architecture decision

HA's premise is that a guest's disks outlive the node it ran on. Three ways to get there:

Ceph: hyperconverged, the full answer

Proxmox integrates Ceph natively: each node contributes disks (OSDs), Ceph replicates data across nodes (3 copies by default), every node sees all storage, and losing a node loses nothing. Self-healing, no external SAN, scales by adding nodes — this is the canonical hyperconverged cluster, and the official requirements are honest about its appetite: three-plus preferably identical servers, a dedicated 10 Gbit/s+ network exclusively for Ceph (25 Gbit/s+ where NVMe is involved — a single modern NVMe can saturate 10 Gbit/s by itself), disks on HBAs, not RAID controllers (Ceph does the redundancy; a RAID layer underneath actively hurts), roughly a CPU thread per Ceph service and ~8 GiB RAM per OSD for comfort. Budget those resources as infrastructure overhead on each node, on top of guest sizing.

ZFS replication: the pragmatic middle

Where Ceph's requirements outrun the budget, pvesr replicates guest volumes between nodes on a schedule — as tight as every minute — over the migration network, ZFS-to-ZFS. Failover then restarts the guest from its replica with one caveat Ceph doesn't have: an explicit data-loss window — whatever changed since the last sync (up to a minute, or your schedule) is gone. For many workloads that trade is excellent: no shared-storage complexity, local-NVMe speed for every guest, redundancy per-guest rather than cluster-wide. Requires ZFS-backed guest storage on both ends.

External shared storage

NFS or iSCSI reachable by all nodes also satisfies HA — simplest conceptually, and it relocates the problem: that storage box is now the single point of failure your cluster doesn't cover. Sensible when it's a genuinely resilient storage system; a false economy when it's one NFS server on the same rack.

What failover actually looks like

The mechanism deserves demystifying, because "the cluster handles it" hides one spicy behaviour. Each node running HA guests arms a watchdog — a dead-man's switch (hardware watchdog where configured, kernel softdog otherwise). When a node loses quorum, it stops petting the watchdog, and after the timeout (~60 seconds) the node fences itself: it reboots, guaranteeing its guests are truly dead and their disks released. The quorate majority waits out that window, then restarts the fenced node's HA guests from shared/replicated storage.

Total experienced downtime for a guest: watchdog window + scheduling + guest boot — a few minutes, typically. Configuration is per-guest (Datacenter → HA): mark the guests that warrant it, group/affinity rules steer where they land (PVE 9 added proper resource-affinity rules, and 9.2's dynamic load scheduler can balance by real-time usage). Two operational notes that prevent self-inflicted incidents: use maintenance mode (and PVE 9.2's cluster-wide HA disarm) before deliberate reboots so the watchdog machinery doesn't "recover" a node you're patching; and know that fencing is also why you never fake quorum on a partitioned node — the watchdog is the last line preventing two copies of your database writing to one disk.

When not to cluster

• Two nodes, no third vote anywhere. Add a QDevice (any small VM elsewhere) or stay uncluster'd — a two-node cluster without one fails worse than independent nodes.
• Nodes separated by real latency. >5 ms means no. Multi-site resilience is solved with independent sites plus replication and backups, not a stretched cluster.
• When the honest requirement is "restore by tomorrow". One well-backed-up host with sub-minute reprovisioning behind it is dramatically less to operate, and PBS live-restore narrows the gap further.
• Mismatched hardware for Ceph. Wildly different nodes make Ceph unhappy (slowest disk sets the pace) — ZFS replication tolerates asymmetry far better.

Frequently asked questions

Can I build a two-node cluster?

Yes, with the QDevice as the mandatory third vote — corosync-qnetd on any always-on machine outside the pair (a small cloud instance is the classic choice; it runs no guests and needs trivial resources). With it, either node can fail and the survivor keeps quorum and runs HA recovery. Without it, don't: any link or node failure freezes the whole cluster, which is strictly worse than two independent hosts.

Does HA give me zero-downtime failover?

No — and knowing why sets correct expectations. Unplanned failover is fence-then-restart: the failed node's guests boot afresh on survivors after the watchdog window, so plan for minutes of downtime and design applications to recover cleanly from a hard stop (they should anyway). What is zero-downtime is planned movement: live migration drains a node before maintenance with guests running throughout. HA handles the surprises; live migration handles the schedule.

Can my nodes be in different datacenters?

Different racks or rooms in one facility with sub-5 ms private connectivity: yes, and it's good failure-domain hygiene. Different cities: no — corosync membership and (worse) synchronous Ceph replication both degrade into instability over WAN latencies. The multi-region pattern is independent clusters per site with asynchronous replication and backup sync between them, and failover as a deliberate act rather than an automatic one.

Do all nodes need identical hardware?

For the cluster itself, no — mixed nodes cluster fine, and live migration between different CPU generations works with a common baseline CPU type set on guests (the cost: guests can't use newer instructions). For Ceph, near-identical is the strong recommendation — capacity and performance balance across OSDs, and one slow node drags the pool. Pragmatic pattern for mixed fleets: cluster them all for management, run Ceph (or replication pairs) across the matched subset, and pin HA guests there.

Deploying on Serverside

The cluster this article describes maps to a concrete order: three matched dedicated servers — Proxmox pre-installed, NVMe on HBAs for Ceph or ZFS for replication — connected over our private networking for corosync and storage traffic (your public bandwidth never sees cluster chatter), each with KVM-over-IP for the day a node needs out-of-band attention. Provisioning is sub-minute on ASN 55285, which makes even the QDevice pattern trivial: any small third box does.

Build up to it with the series: setup, networking, LXC vs VMs, and backups — which you still need; a cluster replicates your mistakes with excellent availability.