Oracle RAC
Oracle Real Application Clusters (RAC) is an option for the Oracle Database that enables multiple server instances to access a single shared database simultaneously, allowing the database to function as a single logical unit while distributing workload across clustered nodes for enhanced performance and reliability.[1] This architecture contrasts with traditional single-instance Oracle Databases, where only one instance manages the database at a time, by leveraging shared storage and interconnect technology to ensure data consistency and availability.[1] At its core, Oracle RAC relies on Oracle Clusterware as the foundational infrastructure to manage the cluster, binding multiple servers so they operate as a unified system, along with Oracle Automatic Storage Management (ASM) for efficient shared storage handling.[1] Key mechanisms include the Global Cache Service (GCS) and Global Enqueue Service (GES), which facilitate Cache Fusion to transfer data blocks between instances over a private interconnect, maintaining cache coherency without constant disk access.[1] This setup eliminates single points of failure, supports mission-critical applications without requiring application code modifications, and provides scalability by adding or removing nodes dynamically.[1] Oracle RAC delivers high availability through features like automatic failover, transparent application continuity, and zero-downtime rolling maintenance, protecting against hardware failures, software issues, and planned outages.[2] It scales horizontally for diverse workloads, including online transaction processing (OLTP), analytics, AI, and complex enterprise applications such as SAP, Oracle Fusion Applications, and Salesforce, achieving high throughput and ultra-fast response times.[2] Widely adopted in sectors like banking, telecommunications, retail, and government, Oracle RAC ensures 24/7 operation for demanding, always-on environments.[2]Introduction
Definition and Core Concepts
Oracle Real Application Clusters (RAC) is Oracle's shared-everything clustering solution for the Oracle Database, enabling multiple server instances to operate against a single physical database for enhanced scalability and availability.[3] Introduced in 2001 with Oracle Database 9i Release 1, RAC replaced the prior Oracle Parallel Server (OPS), which depended on disk-based locking mechanisms that limited performance in multi-node environments.[4] This architecture allows a cluster of independent servers, or nodes, to function as a unified system, sharing access to all database resources without partitioning data across nodes.[5] The core structure of Oracle RAC centers on shared storage for critical database components, including data files, control files, and redo logs, which are accessible to every node via cluster-aware file systems or storage area networks.[3] Each node runs its own database instance, complete with dedicated memory structures such as the System Global Area (SGA) for caching data blocks and the Program Global Area (PGA) for session-specific operations, along with background processes for local management.[3] Instances coordinate and synchronize through a private, high-speed interconnect network, which facilitates communication for resource allocation and data consistency.[3] In contrast to a single-instance Oracle Database, where a solitary instance handles all database operations, RAC supports active-active scaling across nodes, distributing workload dynamically to improve throughput and resilience without necessitating application code changes.[3] This setup eliminates single points of failure, as the database remains operational on surviving nodes if one fails, with synchronization primarily managed via the Cache Fusion protocol for efficient block transfers between caches.[4]Primary Objectives and Benefits
Oracle Real Application Clusters (RAC) primarily aims to deliver high availability by tolerating node failures without interrupting database operations, enabling horizontal scalability through the addition of cluster nodes, and enhancing performance via parallel query processing across multiple instances.[6] This architecture allows organizations to maintain continuous access to data even during hardware or software failures, supporting mission-critical applications that require uninterrupted service.[2] Key benefits include fault isolation, where a failure on one node does not impact others, ensuring that workloads continue seamlessly on surviving instances; load balancing across nodes to optimize resource utilization and prevent bottlenecks; and integration with Oracle Data Guard for robust disaster recovery, allowing rapid failover to a standby site while maintaining data consistency.[6] These features leverage shared storage for concurrent data access and Cache Fusion for low-latency block transfers between nodes, further minimizing disruptions.[2] In modern versions, such as Oracle Database 23ai (released 2024), Oracle RAC supports clusters with over 100 nodes, achieving near-linear scalability without application modifications, and reduces downtime to under 15 seconds through fast failover mechanisms.[6][7] It is particularly suited for online transaction processing (OLTP) systems demanding 24/7 uptime, such as banking and e-commerce platforms, as well as large-scale analytics workloads that benefit from parallel execution without data partitioning.[2]Prerequisites
Hardware and Software Requirements
Oracle Real Application Clusters (RAC) requires robust hardware configurations to ensure high availability and scalability across multiple nodes. Each node must utilize symmetric multiprocessing (SMP) servers or blade systems capable of supporting the cluster's workload, with a minimum of 8 GB RAM for Oracle Grid Infrastructure installations and at least 1 GB (recommended 2 GB or more) for the Oracle Database software.[8] Shared storage is essential, typically implemented via Storage Area Network (SAN) or Network Attached Storage (NAS) with multipath I/O for redundancy and performance, where Oracle Automatic Storage Management (ASM) is recommended for managing database files, control files, SPFILEs, redo logs, and recovery files.[9] Networking demands at least one public network interface card (NIC) per node for client access and a dedicated private interconnect using high-speed Ethernet, with a minimum of 1 Gbps and recommendations for 10 Gbps or faster to handle Cache Fusion traffic efficiently.[10] On the software side, Oracle RAC necessitates the Enterprise Edition of Oracle Database with the RAC option licensed, alongside Oracle Grid Infrastructure for cluster management, which must be installed in a separate Oracle home from the database software. For Oracle Database 21c and later (including the current 23ai release as of 2025), the multitenant architecture with a container database (CDB) is mandatory.[1][9] Compatible operating systems include certified versions of Linux (such as Oracle Linux 8 and later, Red Hat Enterprise Linux, and SUSE Linux Enterprise Server), Windows Server (e.g., 2019, 2022, 2025), and Oracle Solaris on SPARC or x86-64 platforms, ensuring identical OS configurations across all nodes to avoid compatibility issues.[11][12][13] All hardware and software must comply with Oracle's certification matrices to guarantee support and interoperability. The Oracle Hardware Compatibility List (via My Oracle Support) verifies compatible servers, storage, and network adapters from vendors like IBM, HP, and Oracle/Sun, while the software certification matrix outlines supported OS patches and versions.[11] Virtualization environments are supported, including VMware vSphere, Oracle VM, KVM, and Microsoft Hyper-V, allowing RAC deployment on certified virtual machines with performance considerations for shared resources.[14] The minimum cluster configuration consists of 2 nodes for basic redundancy, with scalability extending to 16 or more physical nodes depending on the storage protocol—such as up to 30 nodes with iSCSI over a private Gigabit Ethernet network—though practical limits are influenced by interconnect bandwidth and storage redundancy.[13][11]Cluster Setup Essentials
Setting up an Oracle Real Application Clusters (RAC) environment requires meticulous planning to ensure high availability, scalability, and performance. Initial planning steps involve conducting a capacity assessment to evaluate workload requirements, including input/output operations per second (IOPS) and throughput needs for the shared storage subsystem, which directly impacts database performance under concurrent access from multiple nodes.[15] Redundancy must be incorporated into the shared storage design, typically using RAID 1+0 configurations to provide both mirroring for fault tolerance and striping for balanced load distribution across disks.[16] Additionally, integrating a backup strategy from the outset is essential, leveraging tools like Recovery Manager (RMAN) to support consistent backups of the shared database while accounting for cluster-wide synchronization.[16] Storage configuration in Oracle RAC centers on Automatic Storage Management (ASM), which simplifies the management of shared disks by creating disk groups that span multiple nodes for pooled storage resources. ASM disk groups are used to store database files, redo logs, and control files, ensuring uniform access and automatic load balancing. Critical components include voting disks, which maintain cluster membership by recording node status—Oracle recommends configuring 3 to 5 voting disks in an ASM disk group with normal or high redundancy to tolerate failures without quorum loss. The Oracle Cluster Registry (OCR) stores cluster configuration data and must be placed in a separate ASM disk group or shared file system, with redundancy provided through mirroring to prevent single points of failure. Note that starting with Oracle Database 21c, third-party clusterware is no longer supported.[1] Network zoning is fundamental to isolate traffic types and enhance security and performance in Oracle RAC. The public network handles client connections and administrative access, requiring gigabit Ethernet or faster links with low latency to support SQL*Net traffic.[17] In contrast, the private interconnect is dedicated to internal cluster communication, such as Cache Fusion for block transfers between instances, and should use a separate high-bandwidth, low-latency network like InfiniBand or 10 Gigabit Ethernet to minimize contention. The private interconnect uses UDP ports (e.g., dynamically assigned in high ranges for Cache Fusion, fixed ports like 1638 for Cluster Synchronization Services). To avoid interference, VLANs are employed for separation, ensuring the private interconnect operates in an isolated segment that prevents public traffic from impacting cluster heartbeat or data synchronization.[18] Security basics in cluster setup focus on node authentication and controlled access to prevent unauthorized participation. The Oracle Advanced Security Option (ASO) enables strong authentication mechanisms, such as Kerberos or PKI-based certificates, to verify node identities during cluster join operations.[19] Firewall configurations must allow specific ports: TCP port 1521 for SQL*Net listener connections from clients, and ensure the private interconnect's UDP ports are open between nodes while blocking external access.[20]Architecture
Key Components
Oracle Clusterware serves as the foundational cluster management software for Oracle Real Application Clusters (RAC), providing a portable infrastructure that binds multiple servers to operate as a single logical system. It manages essential cluster resources, including nodes, virtual IP (VIP) addresses, database services, and listeners, ensuring high availability and automated failover. Key subcomponents include Cluster Ready Services (CRS), which oversees the lifecycle of cluster resources stored in the Oracle Cluster Registry (OCR); Cluster Synchronization Services (CSS), which maintains node membership and synchronizes cluster operations to prevent split-brain scenarios; and Event Management (EVM), which monitors and propagates cluster events for proactive management. These elements collectively enable seamless resource allocation and recovery across the cluster.[3][21] Shared resources form the core data layer in Oracle RAC, allowing multiple database instances to access the same physical storage simultaneously. These include data files, control files, server parameter files (SPFILEs), and redo log files, all residing on cluster-aware shared disks that meet prerequisites such as high-speed access from all nodes. Automatic Storage Management (ASM) instances manage this shared storage, dynamically allocating disk groups and optimizing I/O performance for RAC environments. Oracle Net listeners, configured per node or as shared listeners, facilitate client connections, while the Single Client Access Name (SCAN) enhances scalability by resolving to three IP addresses for connection load balancing and failover across instances.[3][22] Oracle RAC introduces specialized background processes to handle distributed operations, distinguishing it from single-instance databases. Multiple Lock Manager Server (LMS) processes per instance manage buffer cache coordination, including the transfer of data blocks between instances, which is critical for Cache Fusion's reliance on these processes. The Lock Manager Daemon (LMD) process oversees distributed lock management, processing remote enqueue requests, detecting deadlocks, and ensuring resource consistency across the cluster. Additional processes like the Lock Monitor (LMON) support global enqueue monitoring and instance recovery, while the Lock Element (LCK) handles cross-instance calls. These processes operate alongside standard database background processes to maintain data integrity and performance in a multi-node setup.[3] Oracle Grid Infrastructure integrates Oracle Clusterware with ASM, creating a unified platform for automated cluster and storage management in RAC deployments. Installed on each node, it simplifies administration by combining resource management with disk provisioning, supporting up to 128 instances per database and enabling features like dynamic volume management. This integration reduces operational complexity, allowing administrators to treat the cluster as a single entity for provisioning and maintenance tasks.[21][3][23]Cache Fusion Protocol
Cache Fusion is the core protocol in Oracle Real Application Clusters (RAC) that enables efficient synchronization of data blocks across multiple database instances by transferring them directly between buffer caches over the private cluster interconnect, thereby bypassing disk I/O and minimizing latency. This mechanism logically merges the buffer caches of all instances into a single, shared global cache, allowing instances to access data as if it were local without frequent disk contention. By shipping blocks in memory rather than reading from shared storage, Cache Fusion significantly reduces reader/writer conflicts and overall system overhead, enhancing scalability for high-throughput workloads.[3][24][25] The protocol operates in two primary modes to handle read and write requests while maintaining data consistency. For read operations, Cache Fusion transfers consistent read (CR) blocks, which are versions of data blocks constructed to reflect a consistent view at the time of the request, avoiding the need for rollback application on the requesting instance. In write scenarios, it transfers current (dirty) blocks—those modified but not yet written to disk—accompanied by lock mode conversions, such as upgrading from shared (S) to exclusive (X) mode to ensure only one instance can modify the block at a time. These conversions prevent concurrent modifications and preserve block integrity across the cluster.[26][27][25] The Global Cache Service (GCS), a key component integrated with Cache Fusion, manages the modes and locations of data blocks to enforce cache coherency. GCS tracks blocks in modes including null (no holder), shared (multiple readers), exclusive (single modifier), and S/SSX (shared with shared past image for undo reconstruction). It coordinates access privileges by monitoring resource states and facilitating block transfers via Lock Management Services (LMS) processes, which handle inter-instance messaging. Additionally, GCS employs heartbeat mechanisms through these processes to continuously track resource availability and instance liveness, ensuring prompt detection of failures and reallocation of resources.[28][25][29] Performance of Cache Fusion is evaluated using key metrics such asgc cr blocks received and gc cr blocks sent for consistent read transfers, and gc current blocks received and gc current blocks sent for current block transfers, which indicate the volume of inter-instance block traffic. These statistics, available in views like V$SYSSTAT, help identify bottlenecks in global cache activity. Fusion efficiency, often expressed as the ratio of CR blocks to total global cache blocks multiplied by 100, provides a measure of how effectively the protocol avoids disk access, with higher percentages signaling optimal cache utilization.[26][30]
Cache Fusion was introduced in Oracle Database 9i as a groundbreaking feature to enable true shared-cache architecture in RAC, replacing earlier disk-based coordination methods. Subsequent enhancements, particularly from Oracle Database 19c onward, have incorporated support for Remote Direct Memory Access (RDMA) over protocols like RoCE, allowing direct memory-to-memory transfers with reduced CPU involvement and further latency improvements on compatible hardware.[4][31]
Networking and Connectivity
Interconnect Design
The interconnect in Oracle Real Application Clusters (RAC) serves as a dedicated private network that provides a high-bandwidth, low-latency communication pathway between cluster nodes, enabling efficient inter-instance data transfers via Cache Fusion, cluster heartbeats for node membership monitoring, and other essential cluster traffic.[3] This infrastructure is crucial for maintaining data consistency and scalability across multiple database instances, treating the cluster as a unified system.[6] Common implementations utilize 10, 25, or 100 Gigabit Ethernet (GbE) or InfiniBand to meet these performance demands.[3] To ensure high availability, Oracle RAC employs redundancy through dual or multiple interconnect interfaces per node, configured without traditional bonding but leveraging High Availability IP (HAIP) for automatic failover and load balancing.[32] HAIP dynamically assigns virtual IP addresses across available interfaces on different subnets, allowing seamless traffic redirection if a physical link or switch fails, thus preventing single points of failure.[32] The protocol stack primarily uses UDP for its efficiency in handling the bursty, high-volume traffic patterns, with support for Reliable Datagram Sockets (RDS) over InfiniBand as an alternative.[3] Bandwidth requirements for the interconnect start at a minimum of 1 Gb/s, but Oracle recommends 10 Gb/s or higher to support intensive workloads without bottlenecks.[33] Latency must be kept low, ideally under 1 ms round-trip, to optimize Cache Fusion performance and minimize global cache waits.[34] These specifications ensure that the interconnect functions effectively as a shared memory channel equivalent for the distributed database environment.[6] Configuration of the interconnect involves assigning a private IP address range exclusive to cluster communication, separate from public networks, to isolate and prioritize internal traffic.[3] Enabling jumbo frames with a Maximum Transmission Unit (MTU) of 9000 bytes is recommended across all components, including host adapters, drivers, and switches, to reduce overhead and improve throughput for large block transfers.[33] Starting with Oracle Database 12c, integration with RDMA over Converged Ethernet (RoCE) enhances performance by enabling kernel-bypass direct memory access, particularly in environments supporting RDMA protocols like RDSv3.[13]Name Resolution and VIPs
In Oracle Real Application Clusters (RAC), name resolution for client connections occurs over the public network, where clients access database instances without needing to know specific node details. This setup relies on virtual IP addresses (VIPs) and domain name system (DNS) configurations to ensure seamless connectivity and high availability. Node VIPs are floating IP addresses hosted by Oracle Clusterware on the public network interface of each cluster node, allowing clients to connect directly to an instance via these addresses.[35] By using VIPs, Oracle RAC achieves sub-second failover for client connections; if a node fails, the VIP migrates to another available node, preventing the typical TCP timeout delays of up to several minutes that would occur with static IP binding.[36] To simplify client access and enable load balancing across the cluster, Oracle RAC introduced the Single Client Access Name (SCAN) in release 11g. SCAN is a DNS-based alias that resolves to three IP addresses, each associated with a SCAN VIP and listener, distributing incoming connections evenly without requiring clients to specify individual node VIPs.[37] These SCAN VIPs function similarly to node VIPs but can relocate to any node in the cluster, providing a stable, single endpoint for clients regardless of node additions or failures.[38] The SCAN configuration enhances scalability, as it supports up to three addresses for redundancy and load distribution, and integrates with the cluster's public network zoning to maintain consistent access.[39] For automated management of VIPs and hostnames in larger or dynamic environments, Oracle RAC introduced the Grid Naming Service (GNS) in release 11g Release 2 (11.2). GNS operates as a cluster-managed DNS service, using a static GNS VIP address delegated by the external DNS to handle dynamic resolution of cluster resources, including node VIPs and SCAN addresses.[40] This integration reduces administrative overhead by automatically updating DNS records for VIP migrations during node additions, removals, or failures, ensuring clients always resolve to active resources without manual intervention.[41] Client applications connect to Oracle RAC using standard protocols like JDBC or ODBC, configured to target the SCAN name for transparent node affinity and failover. For instance, a JDBC connection string might specify the SCAN hostname and port, allowing the driver to select an optimal instance based on service preferences while receiving Fast Application Notification (FAN) events for real-time updates on cluster changes.[37] FAN, part of the Oracle RAC high availability framework, notifies applications via APIs or callbacks about service relocations or instance failures, enabling proactive reconnection without full session loss.[42] This configuration supports ODBC drivers similarly, promoting workload balancing and rapid recovery in enterprise deployments.[43]Implementation and Operations
Deployment Process
The deployment of Oracle Real Application Clusters (RAC) begins with preparing the operating system on all cluster nodes to meet the necessary prerequisites, such as installing required kernel packages, configuring user accounts like the grid and oracle owners, setting kernel parameters for shared memory and semaphores, and ensuring SSH equivalence for passwordless communication between nodes. These steps ensure compatibility and secure inter-node operations before proceeding to software installation.[44] Next, Oracle Grid Infrastructure is deployed using the Oracle Universal Installer (OUI), which is run on the first node in graphical or silent mode to install Oracle Clusterware and Oracle Automatic Storage Management (ASM) across the cluster.[45] During the installation, OUI prompts for cluster node selection, virtual IP addresses, and storage options; it then copies the software to remote nodes and requires execution of theroot.sh script on each node as the root user to configure system-level components like the Cluster Ready Services (CRS).[46] Following Grid Infrastructure installation, ASM disk groups are created using the ASM Configuration Assistant (ASMCA) or through OUI options to provision shared storage for the database files, voting disks, and OCR.
The Oracle Database software is then installed in RAC mode using OUI in "software only" configuration, selecting the Enterprise Edition and specifying the cluster nodes, with Oracle base and home directories distinct from the Grid home.[47] After software installation, the root.sh script is run on all nodes if not automated during OUI. To create the RAC database, the Database Configuration Assistant (DBCA) is invoked from the Oracle home, choosing the "Oracle Real Application Clusters database" template, specifying instances per node, and configuring parameters like memory and storage allocation to ASM disk groups. DBCA automates the creation of database instances, services, and listeners across nodes.
Cluster resources, including databases and instances, are managed post-deployment using the Server Control (SRVCTL) utility, which allows starting, stopping, and monitoring RAC components from any node.[3] For maintenance, Oracle RAC supports rolling upgrades through a two-stage process: first, upgrading the Grid Infrastructure software node-by-node while maintaining cluster availability, followed by applying patches or database upgrades in batches to minimize downtime.[48] This approach leverages QUORUM-based voting disk management, where a majority of voting disks must remain accessible to sustain cluster operations during the upgrade.[48]
Verification of the deployment ensures all components are operational; the Cluster Verification Utility (CLUVFY) is run with commands like cluvfy stage -post crsinst -n all to check post-installation status of Clusterware and interconnect.[49] Additionally, crsctl check crs confirms the status of Cluster Ready Services, CSS, and EVM daemons on each node, while srvctl status database -d db_name verifies database instance availability across the cluster.[50] These tools provide comprehensive diagnostics to validate the RAC environment before production use.
Management and Diagnostics
Oracle Real Application Clusters (RAC) management involves a suite of command-line utilities and repositories designed to administer cluster resources, instances, and services efficiently. The Server Control Utility (SRVCTL) serves as the primary interface for managing Oracle RAC databases, enabling administrators to start, stop, and relocate database instances, services, and other cluster resources from a centralized command line. For instance, commands likesrvctl start database -d db_name initiate all instances of a specified database across the cluster, while srvctl stop instance -d db_name -i instance_name halts a specific instance without affecting others.[51] Complementing SRVCTL, the Cluster Ready Services Control (CRSCTL) utility provides control over Oracle Clusterware components, allowing operations such as checking cluster status with crsctl check cluster -all, starting or stopping the entire cluster stack via crsctl start crs, or managing high-availability resources.[52] These tools ensure seamless administration of multi-node environments by abstracting complex cluster interactions into simple, scriptable commands.
The Grid Infrastructure Management Repository (GIMR) enhances diagnostics by storing cluster-wide operational data, including performance metrics, alerts, and historical logs from Oracle Clusterware and RAC components, in a dedicated multitenant database pluggable database (PDB) per cluster. Administrators access GIMR data through tools like Oracle Enterprise Manager or SQL queries to diagnose issues such as resource failures or configuration drifts, facilitating proactive maintenance without relying on external storage.
Monitoring in Oracle RAC leverages Automatic Workload Repository (AWR) and Automatic Database Diagnostic Monitor (ADDM) to capture and analyze cluster-specific performance data, with a focus on global cache (gc) wait events that indicate inter-node block transfer delays. AWR snapshots, generated hourly by default, collect statistics on gc cr block receive time (waits for current or consistent read blocks) and gc buffer busy waits (contention for cached blocks), enabling identification of interconnect bottlenecks or load imbalances across instances. ADDM processes these AWR snapshots to produce actionable recommendations, such as optimizing SQL statements contributing to excessive gc waits or adjusting instance parameters for better cache fusion efficiency.[53][54] For real-time visualization, Oracle Enterprise Manager (OEM) offers dashboards that monitor RAC clusters at the node, instance, and service levels, displaying metrics like CPU usage, I/O throughput, and cluster interconnect health through graphical interfaces and alerts. OEM integrates with Clusterware to provide end-to-end views, including proactive notifications for threshold breaches.[55]
Diagnostics in Oracle RAC emphasize automated collection and analysis to expedite issue resolution. The Trace File Analyzer (TFA) automates the gathering of diagnostic logs, traces, and system information from Grid Infrastructure, RAC databases, and supporting OS components, supporting proactive detection of problems like memory leaks or network faults via its collector daemon. Administrators invoke TFA with commands such as tfactl diagcollect to bundle relevant files for support analysis. Oracle Recommended Patch Analyzer (ORAchk) performs comprehensive health checks, scanning configurations, patches, and best practices compliance across the RAC stack to identify vulnerabilities or misconfigurations before they impact availability. It generates reports with severity ratings and remediation steps, often run periodically or on-demand via orachk -runallchecks.[56] Event handling is streamlined through Fast Application Notification (FAN) and Fast Connection Failover (FCF), where FAN notifies applications of cluster events like node failures or service relocations, and FCF enables connection pools to automatically redirect to surviving instances in under a second, minimizing downtime without application code changes.[57]
Common issues in Oracle RAC, such as split-brain scenarios where network partitions lead to multiple nodes attempting concurrent database control, are resolved through Oracle Clusterware's fencing mechanisms, which evict errant nodes via node kill or power-off actions to maintain data integrity and prevent corruption. Interconnect latency, often manifesting as elevated gc waits, is diagnosed using standard OS utilities like ping for round-trip times and traceroute for path analysis on the private cluster network, helping isolate hardware or configuration problems.[58]