Oracle Cluster

Introduction

Oracle Real Application Clusters (RAC) Multi-node Cluster is a database architecture that runs a single Oracle database across multiple servers (nodes), enhancing availability, scalability, and performance by distributing the workload and ensuring continuous operation even if one node fails.

Key Features

• High Availability (HA): Continuous operation by eliminating single points of failure; nodes continue to operate if one fails.
• Scalability: Add more nodes to handle increased users and transactions.
• Load Balancing: Even distribution of workloads across nodes, optimizing performance.
• Fault Tolerance: Redundancy ensures database availability despite node failures.
• Improved Performance: Multiple nodes handle more transactions and queries.

Components of Oracle RAC Multi-Node Cluster

• Cluster Nodes: Servers running an instance of the Oracle Database.
• Oracle Clusterware: Software for clustering services, including crsctl and srvctl.
• Shared Storage: Nodes share access to storage via SAN, NAS, or Oracle ASM.
• Interconnect: Private network for internode communication.
• Oracle ASM: Simplifies storage management, providing striping and mirroring.
• Global Resource Directory (GRD): Tracks data blocks and resources across instances.

Oracle Data Guard in RAC

Oracle Data Guard enhances RAC’s high availability, data protection, and disaster recovery by maintaining synchronized standby databases.

Key Features

• Disaster Recovery: Standby databases in different locations for site-level recovery.
• Data Protection: Continuous application of redo logs ensures data consistency.
• High Availability: Handles node-level and site-level failures for robust availability.

Key Benefits

• Unified Device Discovery: Provides a comprehensive view of all elements in an Oracle RAC Database with Data Guard Multi-Node Cluster, including their relationships.
• Proactive Device Monitoring: Collects metric values over time and sends alerts to the appropriate team when thresholds are breached or unexpected behavior occurs, ensuring minimal or zero downtime.
• Job Scheduling Metrics: Offers detailed metrics on job scheduling times and statuses.
• Concern Alerts: Generates alerts for each metric to notify administrators of any resource issues promptly.

Supported Target Versions

The application is validated on Oracle Database 19c Enterprise Edition Release 19.0.0.0.0.

Prerequisites

• OpsRamp Classic Gateway 15.0.0 and above.

• OpsRamp Nextgen Gateway 15.0.0 and above Note: OpsRamp recommends using the latest Gateway version for full coverage of recent bug fixes, enhancements, etc.

• For Bash CLI cmdlets the following are prerequisites:

  • SSH User (Prefer - Oracle user) should be able to execute bash commands and listener related commands (like crsctl, srvctl..etc).

• Users can establish db connection from gateway to oracle scan name and local listeners as well.

Oracle authorization permissions:

For monitoring some metrics, we are using JDBC. For JDBC connections we are supporting Database authentication.

This utilizes CLI commands such as crsctl, srvctl, and olsnodes..etc for monitoring and discovery. Additionally, we are not using .oraenv to set the Oracle environment; instead, we configure the Oracle environment variables in .bashrc.

Please find the below screenshot having oracle environment configuration in .bashrc file

Please check below points in gateway:

• ping <scan name> - scan hostname / scan ip address based on what is provide in the configuration. (If you are using scan hostname, ensure that the hostname is resolved by checking proper dns is configured on the gateway.)
• telnet <scan name> 1521
• connect to gcli using “gcli” cmd
• execute
  db oracledb <scan_name> <username> <password> <db_port> <db_name>:servicename 15000 10000 insecure Yes "SELECT INST_ID, INSTANCE_NUMBER, INSTANCE_NAME, HOST_NAME FROM gv$instance"

Note: While establishing connection on the scan hostname / Ipaddress it is internally redirected to the local listeners, ensure that the end device (all RAC nodes) accepts inbound connections on all these IpAddresses.

Privileges - The provided database user should have the SELECT ANY TABLE privilege.

Roles - The provided database user should have the CONNECT and SELECT_CATALOG_ROLE

Hierarchy of Oracle RAC resource

      • Oracle RAC
             - Oracle Node
                    - Oracle DB Instance
             - Oracle Disk Group
                    - Oracle Disk

Supported Metrics

Default Monitoring Configurations

Oracle RAC Multi Node Cluster application has default Global Device Management Policies, Global Templates, Global Monitors and Global Metrics in OpsRamp. You can customize these default monitoring configurations as per your business requirement by cloning respective Global Templates and Global Device Management Policies. It is recommended to clone them before installing the application to avoid noise alerts and data.

1. Default Global Device Management Policies

   You can find the Device Management Policy for each Native Type at Setup > Resources > Device Management Policies. Search with suggested names in global scope:

   {appName nativeType - version}

   Ex: oracle-cluster Oracle RAC - 1 (i.e, appName = oracle-cluster, nativeType = Oracle RAC, version = 1)

2. Default Global Templates

   You can find the Global Templates for each Native Type at Setup > Monitoring > Templates. Search with suggested names in global scope. Each template adheres to the following naming convention:

   {appName nativeType 'Template' - version}

   Ex: oracle-cluster Oracle RAC Template - 1 (i.e, appName = oracle-cluster, nativeType = Oracle RAC, version = 1)

3. Default Global Monitors

   You can find the Global Monitors for each Native Type at Setup > Monitoring > Monitors. Search with suggested names in global scope. Each Monitors adheres to the following naming convention:

   {monitorKey appName nativeType - version}

   Ex: Oracle RAC Monitor oracle-cluster Oracle RAC 1(i.e, monitorKey = Oracle RAC Monitor, appName = oracle-cluster, nativeType = Oracle RAC, version = 1)

Configure and Install the Oracle Cluster Integration

1. To select your client, navigate to All Clients, and click the Client/Partner dropdown menu.
   Note: You may either type your client’s name in the search bar or select your client from the list.
2. Navigate to Setup > Account. The Account Details screen is displayed.
3. Click Integrations. The Installed Integrations screen is displayed with all the installed applications.
   Note: If you do not have any installed applications, you will be navigated to the Available Integrations and Apps page with all the available applications along with the newly created application with the version.
4. Click + ADD on the Installed Integrations page.
   Note: Search for the integration either by entering the name of the integration in the search bar or by selecting the category of the integration from the All Categories dropdown list.
5. Click ADD in the Oracle Cluster application.
6. In the Configuration screen, click + ADD. The Add Configuration screen appears.
7. Enter the following BASIC INFORMATION:

8. CUSTOM ATTRIBUTES: Custom attributes are the user-defined data fields or properties that can be added to the preexisting attributes to configure the integration.

Note: The custom attribute that you add here will be assigned to all the resources that are created by the integration. You can add a maximum of five custom attributes (key and value pair).

9. In the RESOURCE TYPE section, select:
   • ALL: All the existing and future resources will be discovered.
   • SELECT: You can select one or multiple resources to be discovered.
10. In the DISCOVERY SCHEDULE section, select recurrence pattern to add one of the following patterns:
    • Minutes
    • Hourly
    • Daily
    • Weekly
    • Monthly
11. Click ADD.

Now the configuration is saved and displayed on the configurations page after you save it.
Note: From the same page, you may Edit and Remove the created configuration.
12. Under the ADVANCED SETTINGS, Select the Bypass Resource Reconciliation option, if you wish to bypass resource reconciliation when encountering the same resources discovered by multiple applications.
Note: If two different applications provide identical discovery attributes, two separate resources will be generated with those respective attributes from the individual discoveries.
13. Click NEXT.
14. (Optional) Click +ADD to create a new collector. You can either use the pre-populated name or give the name to your collector.
15. Select an existing registered profile.

16. Click FINISH.
    The integration is installed and displayed on the INSTALLED INTEGRATION page. Use the search field to find the installed integration.

Modify Oracle Cluster Integration

• See Modify an Installed Integration or Application article.

  Note: Select Oracle Cluster.

Discover Resources in Oracle Cluster Integration

1. Navigate to Infrastructure > Search > DATABASES > Oracle Cluster. The Oracle Cluster page is displayed.
2. Select the application on the Oracle Cluster page
3. The RESOURCE page appears from the right.
4. Click the ellipsis (…) on the top right and select View Details.

5. Navigate to the Attributes tab to view the discovery details.

View resource metrics

To confirm Oracle Cluster Cluster monitoring, review the following:

• Metric graphs: A graph is plotted for each metric that is enabled in the configuration.
• Alerts: Alerts are generated for metrics that are configured as defined for integration.

6. Click the Metrics tab to view the metric details for Oracle Cluster.

Supported Alert Custom Macros

Customize the alert subject and description with the following macros so that it can generate alerts accordingly.

Supported macros keys:

Resource Filter Input keys

Oracle Cluster application Resources are filtered and discovered based on below keys.

Note: You can filter the resources with the discoverable keys only.

The following tabs represent the Resource Type of Oracle Cluster

Risks, Limitations & Assumptions

• The integration can manage critical/recovery failure alerts for the following two scenarios when the user activates App Failure Notifications in the settings:
  • Connectivity Exception
  • Authentication Exception
• Oracle Cluster will send any duplicate/repeat failure alert notification for every 6 hours.
• Metrics can be used to monitor Oracle resources and can generate alerts based on the threshold values.
• We have provided the provision to give Cluster Ip Address OR HostName in configuration, but hostName provision will work only if the host name resolution works.
• Oracle Cluster supports only Classic Gateway and NextGen Gateway. Not supported with Cluster Gateway.
• No support of showing activity logs.
• The Template Applied Time will only be displayed if the collector profile (Classic and NextGen Gateway) is version 18.1.0 or higher.
• Component level thresholds can be configured on each resource level.
• Latest snapshot metric support from Gateway 14.0.0.

Troubleshooting

Before troubleshooting, ensure all prerequisites prerequisites are met.

If Oracle Cluster integrations fail to discover or monitor, troubleshoot using the following steps:

• Check if any alerts have been generation on the cluster or in vprobe.
• If there is an error or alert related to the end device connectivity or authentication, try checking the reachability of the end device from the gateway with the following commands:
  • to ping the scan hostname provided in the configuration: {ping <scan name>}
  • to try telnet: {telnet <scan name> <Port>}
  • to try ssh to the end device: {ssh <username>@<node IP Address>}
  • to connect to the gcli: {gcli} {db oracledb <scan_name> <username> <password> <db_port> <db_name>:servicename 15000 10000 insecure Yes "SELECT INST_ID, INSTANCE_NUMBER, INSTANCE_NAME, HOST_NAME FROM gv$instance"}
  • to try SSH to nodes: {ssh <user name>@<node IPAddress>}

Version History