powershelldba.de

Quorum and Witness: Why Failover Fails and How to Fix It

Your cluster works fine until it doesn't. A node goes down, another loses network, and suddenly your AlwaysOn cluster stops responding. The culprit isn't always hardware—it's quorum. And the fix isn't always a reboot.

What Is Quorum?

In a Windows Failover Cluster, quorum is a simple rule: the majority of cluster nodes must be up and communicating to keep the cluster running.

This rule exists to prevent split-brain scenarios—situations where the cluster partitions into isolated groups, each thinking it's the legitimate cluster. Without quorum rules, you could end up with two "primaries" running simultaneously, corrupting data.

Quorum Rules

Notice the pattern: you need a strict majority (51%+) of votes to form quorum.

The Witness Node: Breaking the Tie

A witness is a tiebreaker—it's a light-weight node (physical server, VM, or cloud instance) that participates in quorum voting but does not host SQL Server or cluster resources. It exists only to vote.

Why Witnesses Matter

In a 2-node cluster without a witness:

Node1: UP    Node2: DOWN
Votes:  1          0
Result: 50% → NO QUORUM → Cluster STOPS

In a 2-node cluster with a witness:

Node1: UP    Node2: DOWN    Witness: UP
Votes:  1          0             1
Result: 66% → QUORUM ACHIEVED → Cluster CONTINUES, Node1 owns resources

The witness "tips the scales" in favor of the surviving node.

Witness Types

Witness Type How It Works Best For
File Share Witness (FSW) A shared network file path (SMB). Cluster writes a small marker file; nodes read it to confirm witness presence. Traditional 2-node clusters, LAN-based deployments. No quorum disk required.
Disk Witness A shared SAN LUN or CSV (Cluster Shared Volume). Nodes read/write a tiny control file on the LUN. SAN-attached clusters. Lower latency than FSW; risk of SAN failure affecting all nodes.
Cloud Witness (Azure) A tiny file stored in Azure Blob Storage. Nodes authenticate and read/write status to the cloud. Hybrid/cloud clusters, disaster recovery across geographies. Requires Azure subscription and internet access.
No Witness (Node Majority) Only cluster nodes vote. No external tiebreaker. 3+ node clusters where you can guarantee an odd number. Simple, no external dependencies.

Real-World Failure Scenarios

Scenario 1: A Node Fails—Cluster Stops Anyway

You have a 2-node cluster with a witness. Node1 is the SQL primary. Node2 goes down unexpectedly.

Expected: Node1 stays up; cluster continues; SQL AlwaysOn fails over automatically (if configured).
Actual: The entire cluster stops responding.

Why? The witness is unreachable. Network blip? Witness server is down? File share is offline? Any of these prevent Node1 from confirming the witness vote.

Fix:

-- Check cluster status (run on any cluster node)
Get-ClusterQuorumConfiguration

-- Check witness accessibility (from a cluster node):
Test-Path \\WitnessServer\ShareName\cluster.ini

-- Manually force quorum if witness is truly gone (dangerous!):
Start-ClusterNode -ClearQuarantine

Scenario 2: Network Partition (Split Brain)

You have a 3-node cluster (Node1, Node2, Node3). Node1 and Node2 lose network connectivity to Node3, but can talk to each other.

Without quorum protection: Both partitions might try to run as independent clusters, each hosting resources and running SQL databases. Disaster.
With quorum: Partition A (Nodes 1+2 = 66%) keeps cluster running. Partition B (Node 3 alone = 33%) stops itself to prevent split-brain.

This is quorum working as designed—sacrificing availability to protect data consistency.

Scenario 3: Witness Disk Fails (Disk Witness Only)

Your cluster uses a disk witness on a SAN. The SAN LUN goes offline or becomes inaccessible.

2-node cluster: Immediately loses quorum. Entire cluster stops. (That's why 2-node clusters with disk witnesses are high-risk.)
3-node cluster: No immediate impact if nodes can talk to each other. Cluster continues with node-majority quorum. Witness is simply unavailable until SAN recovers.

Lesson: Disk witnesses introduce a single point of failure. File share witnesses are more resilient because they're network-based and usually redundant.

Diagnosing Quorum Issues

Check Current Quorum Configuration

$cluster = Get-Cluster
$cluster | Get-ClusterQuorumConfiguration

# Output example:
# QuorumType       : NodeAndDiskMajority
# QuorumResource   : Cluster Disk 1
# QuorumThreshold  : 2  (means majority vote requirement)

Check Node Status and Votes

Get-ClusterNode | Select-Object -Property Name, State, NodeHighestVersion

# Output:
# Name   State        NodeHighestVersion
# Node1  Up           6
# Node2  Up           6
# Node3  Paused       6
#
# "Up" = voting, "Paused" = not voting (intentional)
# "Down" = offline or unreachable

Check Witness Accessibility

-- For File Share Witness
Test-Path \\WitnessServer\ClusterWitness

-- For Disk Witness
Get-ClusterResource | Where-Object { $_.ResourceType -eq 'Physical Disk' }

Check Cluster Events (Event Viewer)

On any cluster node, open Event Viewer → Windows Logs → System. Filter for source = "Cluster". Look for:

Recovering from Quorum Loss

Safe Recovery: Force Quorum on Surviving Nodes

If your cluster has lost quorum but you have surviving nodes, you can force one node to restart the cluster without waiting for quorum.

⚠ WARNING: Only use this when you're sure the isolated partition is truly down and won't rejoin. Forcing quorum on two separate partitions causes split-brain and data corruption.
-- On the surviving node you want to "own" the cluster:
Stop-ClusterNode -Name "LocalNodeName"
Start-ClusterNode -ClearQuarantine

# This removes the local node from quorum voting temporarily,
# restarts it, and resynchronizes with other surviving nodes.

If Witness Is Down Permanently

-- Remove the failed witness and reconfigure quorum
$cluster = Get-Cluster
$cluster | Remove-ClusterQuorum
$cluster | Set-ClusterQuorumConfiguration -NodeAndFileShareMajority \\NewWitnessServer\NewShare

# Or, if you have 3+ nodes, switch to node-majority (no witness):
$cluster | Set-ClusterQuorumConfiguration -NodeMajority

Partial Cluster Recovery

If multiple nodes are down and you need to restore cluster functionality:

-- Bring up one node completely
Start-ClusterNode -Name "Node1"

-- Wait for it to stabilize (30–60 seconds)
Get-ClusterNode | Where-Object { $_.State -eq 'Up' }

-- Bring up other nodes one at a time
Start-ClusterNode -Name "Node2"
Start-ClusterNode -Name "Node3"

-- Verify quorum is achieved
$cluster = Get-Cluster
$cluster | Get-ClusterQuorumConfiguration

Quorum Best Practices

For 2-Node Clusters

For 3-Node Clusters

For All Clusters

Tool Support for Cluster Management

Our infrastructure provides automated tools for setting up and managing AlwaysOn and clustering environments. For automated AlwaysOn setup that handles quorum configuration automatically, see our AlwaysOnSetup tool. This tool ensures quorum is properly configured from the start.

The Bottom Line

Quorum is the invisible backbone of Windows Failover Clustering. When it works, you don't think about it. When it fails, your entire cluster stops—even if some nodes are still running.

The key insight: quorum prioritizes data consistency over availability. It will deliberately shut down a cluster partition if it can't guarantee it's the majority, because a split-brain scenario is worse than temporary downtime.

Understand your quorum configuration, test it regularly, and monitor the witness relentlessly. That's the difference between a cluster that fails gracefully and one that fails catastrophically.

← Back to Blog