STONITH
STONITH, an acronym for "Shoot The Other Node in the Head," is a fencing mechanism in high-availability (HA) cluster computing that isolates failed or malfunctioning nodes by powering them off or otherwise preventing them from accessing shared resources, thereby protecting data integrity from corruption caused by rogue nodes or concurrent access.[1][2]
In HA clusters, such as those managed by Pacemaker, STONITH serves as the primary implementation of node-level fencing to mitigate risks like split-brain scenarios, where multiple nodes independently attempt to run the same resources, potentially leading to data inconsistencies or loss.[1][3] When a node becomes unresponsive or fails to confirm the shutdown of its resources, the cluster uses STONITH to ensure safe migration of workloads to healthy nodes, maintaining overall system availability.[4][1]
The process relies on fence devices—external hardware like power distribution units (PDUs), IPMI interfaces, or storage-based death (SBD) mechanisms—and fence agents, software scripts (e.g., fence_ipmilan for IPMI-based fencing) that interface with these devices to execute actions such as powering off or rebooting a target node.[1][5] Configuration in Pacemaker requires enabling STONITH via the stonith-enabled property set to true, along with defining devices and agents tailored to the cluster's environment, including timeouts (e.g., default 60 seconds for reboots) to handle potential delays.[1][2]
STONITH is essential for production HA environments in Linux-based systems, including distributions like Red Hat Enterprise Linux, SUSE Linux Enterprise, and Oracle Linux, where it ensures compliance with quorum rules and prevents cascading failures.[2][3][4] Without proper fencing, clusters risk instability, making STONITH a foundational component for reliable resource management across multi-node setups.[6][1]
Overview
Definition and Purpose
STONITH, an acronym for "Shoot The Other Node In The Head," serves as a critical fencing mechanism in high-availability (HA) cluster systems, designed to forcibly isolate a node suspected of failure by powering it off or disabling its operations.[7] This approach ensures that the node can no longer access shared resources, thereby maintaining the overall stability and reliability of the cluster.[8]
The primary purpose of STONITH is to safeguard data integrity in environments where multiple nodes share storage or resources, such as in Pacemaker-managed clusters. By abruptly terminating the suspected node's activity, STONITH prevents scenarios where a malfunctioning node could continue writing to shared data, leading to inconsistencies or loss.[3] It acts as the implementation of node-level fencing within these systems, guaranteeing that only one node at a time can claim ownership of critical resources.[7]
STONITH addresses the risk of split-brain conditions, a dangerous situation in clustered setups where communication failures cause nodes to lose synchronization and each assumes it is the primary controller of shared resources. In such cases, simultaneous access by multiple nodes could result in data corruption, such as overwriting databases or conflicting file system operations.[8] By enforcing isolation, STONITH mitigates this threat, allowing the surviving nodes to recover safely without ambiguity.[6]
The term STONITH was coined by early developers in open-source cluster projects, like the Linux-HA initiative, as a humorous yet apt description of the mechanism's decisive and irreversible action against errant nodes.[7] This phrasing underscores the necessity of aggressive intervention to preserve cluster coherence, reflecting the challenges faced in pioneering HA technologies.[6]
Role in Cluster Fencing
In high-availability (HA) clusters, fencing is the process of isolating a malfunctioning or unresponsive node to safeguard shared resources, such as storage systems or network fabrics, from potential corruption or unauthorized access.[1] This isolation ensures that only healthy nodes continue operating, preventing scenarios where a failed node could interfere with cluster operations.[9]
STONITH serves as a robust fencing implementation within this framework, distinguished from softer approaches like resource revocation—where access to specific services is merely withdrawn—by enforcing hard isolation through methods such as powering off the node entirely.[1] This definitive action guarantees that the fenced node cannot resume operations without explicit intervention, providing a higher degree of certainty in protecting cluster integrity compared to less aggressive techniques.[10]
In HA environments, STONITH plays a critical role by maintaining quorum among surviving nodes, averting resource conflicts that could arise from concurrent access, and facilitating seamless failover to operational nodes.[1] These benefits are essential for ensuring continuous service availability, as they eliminate the risk of data divergence in shared resources like databases or file systems.[9]
The necessity of STONITH becomes particularly evident in large-scale deployments, where the mean time between failures (MTBF) decreases dramatically as the number of nodes grows—for instance, dropping from days to mere hours in systems with thousands of components—heightening the probability of node malfunctions and communication disruptions.[11] Thus, STONITH's hard fencing is indispensable for scaling HA clusters reliably without compromising data protection.[1]
Technical Principles
How STONITH Operates
STONITH operates as a critical fencing mechanism in high-availability clusters managed by systems like Pacemaker and Corosync, activating upon detection of potential node failures to ensure data integrity. Triggers typically include loss of heartbeat signals from a node or failure to maintain quorum, where Corosync monitors cluster membership and communication via multicast or unicast protocols.[10][1]
The operation follows a structured sequence to isolate the suspect node. First, surviving nodes conduct a quorum vote to confirm the failure and identify the unresponsive node, ensuring majority agreement before proceeding to avoid erroneous actions.[10][12] Second, Pacemaker broadcasts a STONITH request to available fencing resources. Third, the designated agent or device executes the fencing action, such as powering off the node, within a configured timeout period (typically 60 seconds). Finally, the cluster confirms isolation by verifying the node's offline status, often through repeated checks or manual confirmation if automated methods fail.[13][10]
This process specifically addresses failure modes like "zombie" nodes, which may appear failed due to communication issues but continue accessing shared resources, potentially causing data corruption or split-brain scenarios. By forcibly isolating such nodes, STONITH prevents concurrent resource access and maintains exclusive control by healthy cluster members.[12][10]
The decision to invoke STONITH can be represented by a simplified pseudo-algorithm:
If node X is unresponsive for longer than the configured timeout:
If quorum is present and majority agrees on failure:
Initiate fencing of node X via selected device
Else:
Defer action to avoid premature isolation
Confirm fencing success before resuming cluster operations
If node X is unresponsive for longer than the configured timeout:
If quorum is present and majority agrees on failure:
Initiate fencing of node X via selected device
Else:
Defer action to avoid premature isolation
Confirm fencing success before resuming cluster operations
This logic ensures coordinated action only under verified conditions, with retries (default up to two attempts) if initial fencing fails.[13][14]
Integration with Cluster Communication
STONITH operations in Pacemaker-based clusters depend on the Corosync messaging layer for reliable cluster communication, which handles both heartbeat signals to detect node failures and the broadcasting of fencing requests to coordinate actions across nodes. The pacemaker-fenced daemon, responsible for executing STONITH, communicates with peer fencers on other nodes via Corosync to identify available fencing devices and ensure coordinated isolation of failed nodes.[1] This integration allows STONITH to leverage Corosync's low-latency messaging for timely propagation of fencing commands, preventing delays that could lead to data corruption in split-brain scenarios.[10]
Quorum integration is a critical safeguard in STONITH activation, requiring the cluster partition initiating fencing to possess quorum consensus to avoid erroneous actions from minority partitions. Pacemaker enforces this by checking quorum status before proceeding with STONITH, ensuring that only a majority-approved partition can fence nodes and maintain resource exclusivity.[7] Without quorum, fencing is deferred, which helps mitigate false positives triggered by transient network issues or node isolations.[15]
In open-source stacks, Corosync utilizes the Totem protocol over multicast UDP for STONITH request dissemination, enabling efficient, reliable delivery to all cluster members without individual unicast overhead. Default configurations employ UDP port 5405 for this communication, with multicast addresses such as 226.94.1.1 to support group messaging in LAN environments.[16] This protocol ensures ordered and atomic delivery of fencing messages, synchronizing the cluster's view of node states during STONITH events.[17]
If STONITH signals fail due to communication disruptions, such as blocked multicast traffic on port 5405, Pacemaker applies configurable retries (defaulting to two attempts) and timeouts (60 seconds base, extended by 20% for processing) before deeming the operation unsuccessful.[13] In such cases, the cluster may halt resource management on the affected partition or require manual intervention, like external fencing or network restoration, to restore consensus and prevent ongoing risks.[10] This error handling prioritizes safety, often resulting in a quiescent state until the issue resolves.[18]
History and Development
Origins in Linux-HA Projects
STONITH emerged in the late 1990s and early 2000s within the Linux-HA project, a pioneering open-source initiative aimed at providing high-availability clustering for Linux systems. Primarily developed by Alan Robertson at the IBM Linux Technology Center, along with contributions from the broader Linux-HA community, STONITH addressed critical gaps in early cluster management by introducing a mechanism for node isolation without relying on proprietary hardware. The concept was first outlined in discussions surrounding the Heartbeat program, the project's core clustering tool, which sought to enable failover and resource management in distributed environments.[19]
The primary motivation for STONITH stemmed from the vulnerabilities in nascent high-availability setups, where node failures or communication breakdowns could lead to data corruption through concurrent access to shared storage. Traditional quorum-based approaches proved insufficient for guaranteeing exclusive resource ownership, particularly in scenarios involving partitioned or "zombie" nodes that appeared operational but were not. STONITH provided a robust fencing solution by enabling one node to forcibly power off or reset another, ensuring data integrity and preventing split-brain conditions in resource-constrained open-source environments that lacked commercial fencing devices like power distribution units. This approach was emphasized in early design documents as essential for reliable cluster operation.[19]
The acronym STONITH, denoting "Shoot The Other Node In The Head," was first used on the Linux-HA mailing list prior to 2000, gaining traction as a memorable shorthand for the fencing technique.[20] Key milestones included active discussions from 2001 to 2003 on the mailing list, where developers refined the idea, debated implementation challenges, and formalized its role in cluster protocols; for instance, early 2001 threads explored integration with Heartbeat versions like 0.4.9, highlighting issues such as shutdown-triggered fencing. The API design for STONITH was presented by Robertson at the LinuxWorld Conference and Expo in New York from January 30 to February 2, 2001, marking a pivotal step toward standardization and adoption across multiple cluster systems.[21]
First implementations of STONITH appeared in preliminary Heartbeat releases, evolving into a fully pluggable module by Heartbeat version 2.0, which integrated it seamlessly with cluster communication and resource agents around 2005. This modular design supported diverse fencing methods, including UPS-based power control and software-simulated options, allowing flexible deployment in varied hardware setups. By this point, STONITH had become a cornerstone of Linux-HA, influencing subsequent open-source clustering efforts.
Evolution in Modern Cluster Managers
Following the initial development in the Heartbeat project, STONITH transitioned significantly with the emergence of Pacemaker as an independent cluster resource manager. Pacemaker, which originated as a component of Heartbeat, was maintained separately starting around 2008 to support multiple cluster stacks including both Heartbeat and OpenAIS. By the release of Pacemaker 1.0 in 2010, STONITH had evolved into a core resource agent, integrated as stonith-class agents that could be configured, monitored, and managed like other cluster resources, enhancing its reliability and flexibility in high-availability environments.[22][23]
STONITH's standardization accelerated through its adoption in major enterprise distributions and frameworks. It became a standard component in SUSE Linux Enterprise High Availability Extension, where it is used for node fencing in Pacemaker-based clusters. Similarly, Red Hat Enterprise Linux incorporates STONITH as an essential fencing mechanism to prevent data corruption in high-availability setups. Oracle Linux also integrates STONITH for protecting shared data access during node failures. This widespread adoption was further solidified by its inclusion in the Open Cluster Framework (OCF) specifications, which define standards for STONITH agents as extensions of the resource agent API, ensuring interoperability across Linux-based clustering solutions.[24][2][4][25]
In recent years, STONITH has seen enhancements tailored to modern infrastructures, particularly cloud environments. The fence_aws agent, introduced in 2020, enables STONITH to perform fencing via AWS APIs, allowing clusters to isolate failed nodes through cloud provider controls such as EC2 instance termination.[26] Additionally, the Storage-Based Death (SBD) mechanism, a software-based STONITH approach using shared block storage, has benefited from improvements supporting Linux kernel 5.x series, including better multipath handling and watchdog integration for more robust fencing in virtualized and containerized setups. These updates address scalability in hybrid and cloud-native clusters while maintaining data integrity.[27][28]
The ongoing maintenance of STONITH relies heavily on community efforts through ClusterLabs, which hosts the primary repository for fence agents—the software implementations of STONITH devices. This repository, actively updated by contributors, provides a collection of agents compatible with diverse hardware and cloud platforms, ensuring STONITH's adaptability and security in evolving cluster managers. ClusterLabs' stewardship has facilitated contributions from vendors and users, keeping STONITH aligned with contemporary high-availability needs.[29]
Implementation Methods
Hardware-Based Approaches
Hardware-based approaches to STONITH rely on physical devices to isolate failed nodes in a cluster by either cutting power or revoking access to shared resources, ensuring data integrity without software dependencies.[1]
Power management devices form the core of these methods, enabling remote control over a node's power state. The Intelligent Platform Management Interface (IPMI) allows administrators to power cycle nodes via out-of-band management, often integrated with Baseboard Management Controllers (BMCs) such as Dell's iDRAC or HPE's iLO.[30][1] For broader control, Power Distribution Units (PDUs) like APC switched models (e.g., AP7900B) provide outlet-level power switching, supporting multiple nodes from a single rack unit.[30] These devices operate independently of the cluster's primary network and power sources to avoid single points of failure, with Pacemaker agents such as fence_ipmilan for IPMI and fence_apc_snmp for PDUs handling the integration.[1]
Storage fencing complements power methods by isolating nodes from shared storage without full shutdown, using techniques like SAN zoning or LUN masking. In SAN zoning, fiber channel switches reconfigure access rules to exclude the failed node, while LUN masking revokes logical unit number visibility on storage arrays, preventing unauthorized I/O operations.[3] These approaches, often implemented via vendor-specific agents, ensure exclusive resource access and mitigate split-brain risks in environments with persistent storage.[1]
These hardware methods offer high reliability due to their physical enforcement, reducing the chance of incomplete fencing compared to software alternatives, but they demand dedicated infrastructure and initial setup costs.[30][1] Configuration typically involves defining agents in cluster resources with parameters like IP addresses and credentials, followed by testing using tools such as ipmitool for IPMI verification (e.g., ipmitool -I lanplus -H <ipaddr> -U <user> -P <pass> chassis power status) or manual agent execution like fence_ipmilan -a <ipaddr> -l <user> -p <pass> -o reboot.[31][1]
Software-Based Approaches
Software-based approaches to STONITH leverage operating system features, shared resources, or virtual infrastructure APIs to isolate failed nodes without requiring dedicated physical hardware like power switches. These methods are particularly useful in environments where hardware fencing is impractical or cost-prohibitive, such as virtualized setups or clusters relying on commodity storage. However, they often depend on reliable underlying infrastructure, like shared disks or network connectivity, and may require specific kernel modules for enforcement.[1]
One prominent software-based technique is Storage-Based Death (SBD), also known as STONITH Block Device, which uses shared block storage to coordinate fencing actions among cluster nodes. In SBD, nodes communicate by writing messages to reserved slots on a shared disk device, such as a SAN or iSCSI LUN formatted specifically for this purpose (typically 1 MB for 512-byte blocks or 4 MB for 4 KB blocks, supporting up to 255 nodes). The SBD daemon on each node monitors these slots; if a node detects that it has been targeted for fencing—due to a perceived failure or split-brain scenario—it triggers a self-termination action. This is enforced through integration with a watchdog timer: a hardware watchdog or the Linux kernel's softdog module, which the SBD daemon periodically "feeds" to prevent reset; failure to do so causes the node to reboot or halt, ensuring isolation. SBD can operate in diskless mode for clusters with more than two nodes or a quorum device, relying solely on the watchdog without shared storage, though this is unsuitable for simple two-node setups without additional quorum mechanisms.[32]
In virtualized environments, VM-level fencing provides another software-driven STONITH method by utilizing hypervisor APIs to suspend, reset, or migrate virtual machines hosting cluster nodes. For instance, in VMware vSphere clusters, the fence_vmware_soap or fence_vmware_rest agents interact with vCenter Server via SOAP or REST APIs to control VM power states. Configuration involves mapping cluster node names to VM identifiers and authenticating with vCenter credentials; upon fencing, the agent issues commands like "off" or "reboot" to isolate the target VM, preventing it from accessing shared resources. This approach is integrated into Pacemaker clusters through STONITH resource definitions, with testing via commands that list and target specific VMs. Similar agents exist for other hypervisors, such as KVM's fence_virsh or VirtualBox's fence_vbox, enabling node isolation without physical intervention.[33][34]
Network-based software fencing, such as using SSH for remote shutdown or dynamic firewall rules to isolate nodes, represents a less common and generally less reliable approach due to potential vulnerabilities in network paths and the risk of incomplete isolation. For example, the external/ssh STONITH agent allows nodes to execute shutdown commands over SSH with passwordless root access, but it is recommended only for testing as it cannot guarantee reset of truly failed nodes and may lead to data corruption in production scenarios. Methods involving ARP spoofing to disrupt communication or firewall blocks for traffic isolation have been explored but face reliability issues, including dependency on stable network links and susceptibility to bypass, making them unsuitable for critical high-availability setups.[24][1]
Overall, software-based STONITH methods are cost-effective, avoiding the need for specialized hardware, and can be highly effective in controlled environments like clouds or VMs. However, their success hinges on dependencies such as shared storage availability for SBD or hypervisor access for VM fencing, and they often require kernel modules like softdog for robust watchdog enforcement. Limitations include potential single points of failure in storage or networks, necessitating redundant configurations for production use.[6][32]
Configuration and Usage
STONITH Devices and Agents
In Pacemaker clusters, STONITH functionality is implemented through a framework of specialized resource agents known as fence agents, which adhere to the Open Cluster Framework (OCF) standards for STONITH-class resources. These agents abstract the underlying fencing mechanisms, providing standardized commands such as off, reboot, status, and list to interact with hardware or software devices capable of isolating a failed node. The framework allows Pacemaker to configure and manage these agents as cluster resources, with parameters like pcmk_host_map for node-device mapping and stonith-timeout for operation timeouts, ensuring reliable execution without dependency on the cluster's communication layer.[35][10]
STONITH devices and agents are categorized based on their deployment and interaction with the cluster infrastructure, including external, internal, and virtual types. External devices typically involve power distribution units (PDUs) or network-managed switches, such as fence_apc_snmp for SNMP-enabled APC PDUs, which remotely control power outlets to nodes. Internal devices leverage built-in hardware interfaces like Intelligent Platform Management Interface (IPMI), exemplified by fence_ipmilan, which uses IPMI over LAN to power off or reboot servers directly via their baseboard management controllers. Virtual agents target hypervisor environments, such as fence_vmware_soap for VMware ESXi hosts or fence_virt for KVM/QEMU-based virtual machines, enabling soft fencing by suspending or migrating guest nodes without physical power intervention. Additionally, software-based agents like fence_sbd utilize shared storage and watchdog timers to detect and respond to node failures through the SBD (STONITH Block Device) mechanism.[35][10][29]
Selection of STONITH agents depends on the cluster's physical or virtual environment, ensuring the chosen device operates independently of the node being fenced to avoid circular dependencies. For instance, in bare-metal setups with IPMI-enabled servers, fence_ipmilan is preferred for its direct hardware access, while VMware-virtualized clusters require fence_vmware_soap to interface with the vSphere API for guest isolation. Agents must support discovery features like list or static-list via attributes such as pcmk_host_list, and compatibility is verified against supported platforms documented by vendors like Red Hat.[35][10]
Testing STONITH agents employs the stonith_admin tool to validate configurations without triggering actual fencing events, promoting safe verification in production-like setups. Commands include --list-installed to enumerate available agents, --metadata <agent> to inspect parameters and actions for a specific agent like fence_ipmilan, and --reboot <node> with monitoring options to simulate operations. This protocol ensures agent reliability by checking status and metadata before deployment, often complemented by manual script invocation for individual device testing.[35][10]
| Category | Description | Example Agents |
|---|
| External | Power or network switches external to nodes | fence_apc_snmp (PDU), fence_bladecenter (IBM BladeCenter) |
| Internal | Built-in server management interfaces | fence_ipmilan (IPMI), fence_idrac (Dell iDRAC) |
| Virtual | Hypervisor-based isolation | fence_vmware_soap (VMware), fence_virt (KVM/QEMU) |
Practical Setup Examples
In a typical Pacemaker-based high-availability cluster, STONITH can be configured using the crm configure command to define fencing primitives with agents like fence_ipmilan for IPMI-based hardware fencing. For instance, to set up a STONITH device targeting a single node named pcmk-1 via IPMI LAN, the following configuration snippet maps the node's hostname to its IP address and enables monitoring:
primitive Fencing-pcmk-1 stonith:fence_ipmilan \
ipaddr="192.0.2.1" \
lanplus="1" \
login="testuser" \
passwd="abc123" \
pcmk_host_list="pcmk-1" \
op monitor interval=10m timeout=300s
primitive Fencing-pcmk-1 stonith:fence_ipmilan \
ipaddr="192.0.2.1" \
lanplus="1" \
login="testuser" \
passwd="abc123" \
pcmk_host_list="pcmk-1" \
op monitor interval=10m timeout=300s
This primitive uses the pcmk_host_list attribute to associate the cluster node name with the target's IP, ensuring the agent can identify and fence the correct node without requiring pcmk_host_map in this single-target case.[1]
For multi-node setups where the fencing device requires explicit mapping of hostnames to device-specific identifiers (e.g., ports on a PDU), the pcmk_host_map attribute is used in the primitive definition. An example for a two-node cluster using an APC SNMP device fences nodes prod-mysql1 and prod-mysql2 by mapping them to outlet ports 10 and 11:
primitive fence_apc1 stonith:fence_apc_snmp \
ipaddr="198.51.100.1" \
login="fencing" \
passwd="fencing" \
pcmk_host_map="prod-mysql1:10;prod-mysql2:11" \
pcmk_delay_max="8s"
primitive fence_apc1 stonith:fence_apc_snmp \
ipaddr="198.51.100.1" \
login="fencing" \
passwd="fencing" \
pcmk_host_map="prod-mysql1:10;prod-mysql2:11" \
pcmk_delay_max="8s"
Here, pcmk_host_map translates cluster node names to the device's port numbers, allowing the agent to target the appropriate outlet during fencing operations.[1]
Storage-based death (SBD) provides a software-based STONITH method using shared block devices for node isolation, particularly useful in environments with SAN storage. To configure SBD, first initialize the shared disk on one node with appropriate timeouts for message waiting and watchdog operation, such as:
sbd -d /dev/sbd -4 40 -1 20 create
sbd -d /dev/sbd -4 40 -1 20 create
This sets a message wait timeout of 40 seconds (-4) and a watchdog timeout of 20 seconds (-1), tailored for non-multipathed environments to ensure timely detection of failures without premature fencing. Then, edit /etc/sysconfig/sbd on all nodes to specify the device and watchdog:
SBD_DEVICE="/dev/sbd"
SBD_WATCHDOG_DEV="/dev/watchdog"
SBD_WATCHDOG_TIMEOUT="20"
SBD_DEVICE="/dev/sbd"
SBD_WATCHDOG_DEV="/dev/watchdog"
SBD_WATCHDOG_TIMEOUT="20"
The SBD_WATCHDOG_DEV points to the hardware watchdog (e.g., via a kernel module like softdog), while SBD_WATCHDOG_TIMEOUT defines the interval before self-fencing if SBD cannot access the shared device, preventing split-brain scenarios.[5][36]
In a two-node cluster, enabling STONITH requires careful handling to avoid mutual fencing races during simultaneous failures. After defining STONITH primitives, set the cluster property stonith-enabled=true and configure pcmk_delay_max on each fencing resource to introduce a randomized delay up to the specified maximum (e.g., 8 seconds) before executing a fence action:
property stonith-enabled=true
rsc_defaults resource-stickiness=100
property stonith-enabled=true
rsc_defaults resource-stickiness=100
Combined with node priorities (e.g., via location constraints favoring the higher-priority node), pcmk_delay_max staggers fencing attempts, ensuring the lower-priority node delays its action long enough for the higher-priority one to succeed first and prevent both nodes from fencing each other.[1]
To verify STONITH configuration and runtime status, use crm_mon to monitor the overall cluster state, including whether STONITH resources are active and healthy on each node. For detailed STONITH-specific checks, run stonith_admin --list-devices to list configured devices or stonith_admin --status to query the fencing subsystem's operational status, confirming that agents like fence_ipmilan or SBD are responsive.[1]
Challenges and Best Practices
Common Pitfalls
One common pitfall in STONITH deployment is the occurrence of false positives, where temporary network glitches or delays are misinterpreted as node failures, triggering unnecessary fencing actions. This often stems from overly sensitive timeout settings, such as low values for resource operation timeouts or the cluster's failure-timeout meta-attribute, which fail to account for transient issues and lead to premature node isolation.[24][37] To illustrate, if the failure-timeout is not configured to allow fail counts to expire automatically, a single recoverable error can escalate to fencing, disrupting cluster operations without actual failure.[37]
Another frequent issue arises from circular dependencies, particularly in setups where STONITH mechanisms rely on the same network infrastructure used for cluster communication. When network partitions occur, the fencing process may fail because it cannot access the required devices or agents, creating a dependency loop that prevents resolution of the original fault.[38] This is exacerbated in two-node clusters, where bidirectional communication loss can initiate mutual fencing attempts, known as a fence loop, potentially leaving both nodes offline.[38]
While a single STONITH device or method can pose risks by potentially becoming unavailable during outages—such as power loss to a shared PDU or network inaccessibility—Pacemaker mitigates this through fencing topologies that support multiple devices.[39] For instance, network-based fencing agents like those using IPMI over LAN can become unreachable if the management network fails, halting fencing operations and risking split-brain scenarios.[3]
Optimization Strategies
To enhance the reliability of STONITH in Pacemaker clusters, implementing redundancy through fencing topologies allows for multiple STONITH devices per node, configured with priority levels to ensure failover if primary fencing fails. Fencing topologies organize devices into sequential levels based on an index value (typically 1 to 9), where the cluster attempts all devices in the lowest index level first; success requires all devices in that level to complete, but failure triggers progression to higher indices. For example, level 1 might use a direct IPMI device for fast local fencing, while level 2 employs redundant power distribution units (PDUs) as a backup. This setup minimizes single points of failure and is configured using commands like pcs stonith level add <node> <index> devices=<device1>,<device2>.[1][10]
Regular testing of STONITH configurations is essential to verify functionality and prevent issues like false positives during actual failures. Administrators can use automated scripts invoking pcs stonith fence <node> to simulate node fencing in a controlled environment, confirming that the action completes without disrupting healthy resources; for SBD-based setups, fence_tool enables disk-level testing by injecting simulated failures on shared storage to validate watchdog and messaging integrity. These tests should be scheduled periodically, often via cron jobs, to monitor device responsiveness and update configurations as hardware changes. Brief references to common pitfalls, such as network-induced false positives, underscore the need for isolated test environments.[31][36]
In cloud environments like AWS EC2, STONITH adaptations leverage the fence_aws agent to integrate with instance metadata and APIs for reliable fencing without hardware dependencies. This agent uses AWS Identity and Access Management (IAM) roles to query EC2 metadata (e.g., via the Instance Metadata Service at http://169.254.169.254) and execute actions like stopping instances, ensuring split-brain prevention in virtualized clusters. Configuration involves tagging EC2 instances for identification and granting permissions for ec2:StopInstances and related operations, allowing seamless fencing even across availability zones.[40][41]
Performance tuning of STONITH focuses on adjusting the stonith-timeout parameter to align with cluster latency, thereby minimizing downtime while avoiding premature retries. The default timeout of 60 seconds accommodates typical fencing actions, but in low-latency environments, reducing it to 30-40 seconds (based on measured IPMI or API response times) prevents unnecessary delays; conversely, high-latency setups like cross-region clouds may require extensions to 120 seconds. Balancing this with related properties, such as pcmk_off_timeout for device-specific actions, ensures efficient resource recovery without risking incomplete fences.[1][10]