ORAchk Health Checks For The Oracle Stack (ORACHK 2.2.4 and above)

Oracle Database 12c (12.0.1.*) and 11g ( comes with new feature called RACcheck.
Although the RACcheck already exists before these releases. (In 2011 I posted about benefits of RACcheck)

Brief of RACcheck.
RACcheck - The Oracle RAC Configuration Audit Tool
RACcheck is designed to audit vital configuration settings for the Oracle Database, single instance databases, as well as Oracle Real Application Clusters (Oracle RAC) databases. It also includes checks for Oracle Clusterware, Oracle Automatic Storage Management (Oracle ASM) and Oracle Grid Infrastructure.

RACcheck provides best practices recommedations considering the whole stack, including Maximum Availability Architecture (MAA) configurations and is therefore the ideal tool for regular health checks as well as pre- and post-upgrade best practices assessments.

Now Oracle replace/renamed Oracle RACcheck to ORAchk.

ORAchk- Health Checks for the Oracle Stack

ORAchk replaces the popular RACcheck to support a wider range of products. ORAchk version 2.2.4 is now available for download and includes the following key features:

  • RACcheck renamed to ORAchk
  • ORAchk daemon auto-start mode after node reboot (init integration)
  • Merge multiple ORAchk collection reports
  • Upload of installed patches to database
  • Collection Manager for ORAchk, RACcheck and Exachk (Document 1602329.1)
  • ORAchk signature file in /tmp on all nodes to verify last ORAchk run
  • New checks and bug fixes, including
  • 30 Oracle Ebusiness AP module data integrity checks
  • 12 new Database checks
  • 8 new Solaris system checks
  • Supported Platforms

  • Linux x86-64* (Enterprise Linux, RedHat and SuSE 9, SuSE 10 & SuSE 11)
  • Oracle Solaris SPARC (Solaris 10 and 11)
  • Oracle Solaris x86-64 (Solaris 10 and 11)
  • AIX **
  • HPUX**
  • * 32-bit platforms not supported, no planned support for Linux Itanium
    **Requires BASH Shell 3.2 or higher to be installed

    Supported Oracle Releases

  • 10gR2
  • 11gR1
  • 11gR2
  • 12cR1
  • When to Run ORAchk

  • After initial Oracle RAC deployment
  • Before planned system maintenance
  • After planned system maintenance
  • At least once every three months
  • Install/Configure

    It is recommended to run the tool as the database software owner (e.g. oracle). The user may run the tool as the Grid Infrastructure software owner (e.g. grid) and it will collect the same data but database credentials must manually be supplied to perform the database related audit checks. Typically when run as oracle the customer will have OS authentication set up for the oracle database software owner and the database login credentials will not be needed.

    Download ORAchk

    Stage Location:
    It is recommended that the kit be staged and operated from a local filesystem on a single database server in order to provide the best performance possible.

    $ mkdir -p /u01/app/oracle/orachk

    [oracle@node11g01 install]$ cd /u01/app/oracle/orachk

    [oracle@node11g01 orachk]$ unzip orachk.zip
    Archive: orachk.zip
    inflating: raccheck
    inflating: rules.dat
    inflating: collections.dat
    inflating: readme.txt
    inflating: orachk
    creating: .cgrep/
    inflating: .cgrep/ogghc_12101.sql
    inflating: .cgrep/lcgrep4
    inflating: .cgrep/checkDiskFGMapping.sh
    inflating: .cgrep/ogghc_11204.sql
    inflating: .cgrep/lcgreps9
    inflating: .cgrep/ogghc_11203.sql
    inflating: .cgrep/scgrepx86
    inflating: .cgrep/acgrep
    inflating: .cgrep/oracle-upstarttmpl.conf
    inflating: .cgrep/check_reblance_free_space.sql
    inflating: .cgrep/CollectionManager_App.sql
    inflating: .cgrep/exalogic_zfs_checks.aksh
    inflating: .cgrep/hiacgrep
    inflating: .cgrep/init.tmpl
    inflating: .cgrep/lcgreps10
    inflating: .cgrep/preupgrd.sql
    inflating: .cgrep/diff_collections.pl
    inflating: .cgrep/merge_collections.pl
    inflating: .cgrep/ggdiscovery.sh
    creating: .cgrep/profiles/
    inflating: .cgrep/profiles/DA94919CD0DE0913E04312C0E50A7996.prf
    inflating: .cgrep/profiles/D49C0FBF8FBF4B1AE0431EC0E50A0F24.prf
    extracting: .cgrep/profiles/F13E11974A282AB3E04312C0E50ABCBF.prf
    inflating: .cgrep/profiles/EF6C016813C51366E04313C0E50AE11F.prf
    inflating: .cgrep/profiles/D8367AD6754763FEE04312C0E50A6FCB.prf
    inflating: .cgrep/profiles/DF65D6117CB41054E04312C0E50A69D1.prf
    inflating: .cgrep/profiles/EA5EE324E7E05128E04313C0E50A4B2A.prf
    inflating: .cgrep/profiles/E1BF012E8F210839E04313C0E50A7B68.prf
    inflating: .cgrep/profiles/D462A6F7E9C340FDE0431EC0E50ABE12.prf
    inflating: .cgrep/profiles/D49AD88F8EE75CD8E0431EC0E50A0BC3.prf
    inflating: .cgrep/profiles/E2E972DDE1E14493E04312C0E50A1AB1.prf
    inflating: .cgrep/profiles/F32F44CE0BCD662FE04312C0E50AB058.prf
    inflating: .cgrep/profiles/E8DF76E07DD82E0DE04313C0E50AA55D.prf
    inflating: .cgrep/profiles/D49B218473787400E0431EC0E50A0BB9.prf
    inflating: .cgrep/profiles/D49C0AB26A6D45A8E0431EC0E50ADE06.prf
    inflating: .cgrep/profiles/DFE9C207A8F2428CE04313C0E50A6B0A.prf
    inflating: .cgrep/profiles/D49C4F9F48735396E0431EC0E50A9A0B.prf
    inflating: .cgrep/profiles/D49BDC2EC9E624AEE0431EC0E50A3E12.prf
    inflating: .cgrep/profiles/DF65D0F7FB6F1014E04312C0E50A7808.prf
    inflating: .cgrep/scnhealthcheck.sql
    inflating: .cgrep/pxhcdr.sql
    inflating: .cgrep/lcgrep5
    inflating: .cgrep/scgrep
    inflating: .cgrep/raw_data_browser.pl
    inflating: .cgrep/profiles.dat
    inflating: .cgrep/rack_comparison.py
    inflating: .cgrep/versions.dat
    inflating: .cgrep/create_version.pl
    inflating: .cgrep/lcgreps11
    inflating: .cgrep/utluppkg.sql
    inflating: .cgrep/utlusts.sql
    inflating: .cgrep/reset_crshome.pl
    inflating: .cgrep/asrexacheck
    inflating: .cgrep/lcgrep6
    inflating: .cgrep/utlu112i.sql
    inflating: UserGuide.txt

    Running ORAchk Interactively

    [oracle@node11g01 orachk]$ ./orachk

    CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/app/11.2.0/grid?[y/n][y]

    Checking ssh user equivalency settings on all nodes in cluster

    Node node11g02 is configured for ssh user equivalency for oracle user

    Searching for running databases . . . . .

    . .
    List of running databases registered in OCR
    1. dborcl
    2. None of above

    Select databases from list for checking best practices. For multiple databases, select 1 for All or comma separated number like 1,2 etc [1-2][1].1
    . .

    Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS

    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
    Oracle Stack Status
    Host Name CRS Installed ASM HOME RDBMS Installed CRS UP ASM UP RDBMS UP DB Instance Name
    node11g01 Yes Yes Yes Yes Yes Yes dborcl_1
    node11g02 Yes Yes Yes Yes Yes Yes dborcl_2

    Copying plug-ins

    . . . . . . . . . . . . . . . . . . . . . . . . . . .

    . . . . . .

    17 of the included audit checks require root privileged data collection . If sudo is not configured or the root password is not available, audit checks which require root privileged data collection can be skipped.

    1. Enter 1 if you will enter root password for each host when prompted

    2. Enter 2 if you have sudo configured for oracle user to execute root_orachk.sh script

    3. Enter 3 to skip the root privileged collections

    4. Enter 4 to exit and work with the SA to configure sudo or to arrange for root access and run the tool later.

    Please indicate your selection from one of the above options for root access[1-4][1]:- 2

    *** Checking Best Practice Recommendations (PASS/WARNING/FAIL) ***

    Collections and audit checks log file is

    Detailed report (html) - /u01/app/oracle/orachk/orachk_node11g01_dborcl_031814_170552/orachk_node11g01_dborcl_031814_170552.html

    UPLOAD(if required) - /u01/app/oracle/orachk/orachk_node11g01_dborcl_031814_170552.zip

    Below the report generated in this test.
    Oracle RAC Assessment Report

    orachk usage options

    [oracle@node11g01 orachk]$ ./orachk -h
    Usage : ./orachk [-abvhpfmsuSo:c:t:]
    -a All (Perform best practice check and recommended patch check)
    -b Best Practice check only. No recommended patch check
    -h Show usage
    -v Show version
    -p Patch check only
    -m exclude checks for Maximum Availability Architecture (MAA) scorecards(see user guide for more details)
    -u Run orachk to check pre-upgrade or post-upgrade best practices for, and
    -o pre or -o post is mandatory with -u option like ./orachk -u -o pre
    -f Run Offline.Checks will be performed on data already collected from the system
    -o Argument to an option. if -o is followed by v,V,Verbose,VERBOSE or Verbose, it will print checks which passs on the screen
    if -o option is not specified,it will print only failures on screen. for eg: orachk -a -o v

    Pass comma separated node names to run orachk only on subset of nodes.
    Pass comma separated database names to run orachk only on subset of databases
    Run orachk only on local node.
    Run orachk in debug mode. Debug log will be generated.
    eg:- ./orachk -debug
    Skip PASS'ed check to print in orachk report and upload to database.

    Do not print healthscore in HTML report.

    -diff [-outfile ]
    Diff two orachk reports. Pass directory name or zip file or html report file as &

    Compare two different Exalogic rack and see if both are from the same release.Pass directory name or zip file as & (applicable for Exalogic only)

    -c Used only under the guidance of Oracle support or development to override default components

    initsetup : Setup auto restart. Auto restart functionality automatically brings up orachk daemon when node starts
    initrmsetup : Remove auto restart functionality
    initcheck : Check if auto restart functionality is setup or not
    initpresetup : Sets root user equivalency for COMPUTE, STORAGE and IBSWITCHES.(root equivalency for COMPUTE nodes is mandatory for setting up auto restart functionality)

    start : Start the orachk daemon
    start_debug : Start the orachk daemon in debug mode
    stop : Stop the orachk daemon
    status : Check if the orachk daemon is running
    info : Print information about running orachk daemon
    stop_client : Stop the orachk daemon client
    nextautorun : print the next auto run time
    run orachk only if daemon is running
    Dont use daemon to run orachk
    configure orachk daemon parameter like "param1=value1;param2=value2... "

    Supported parameters are:-

    AUTORUN_INTERVAL :- Automatic rerun interval in daemon mode.Set it zero to disable automatic rerun which is zero.

    AUTORUN_SCHEDULE * * * * :- Automatic run at specific time in daemon mode.
    - - - -
    ▒ ▒ ▒ ▒
    ▒ ▒ ▒ +----- day of week (0 - 6) (0 to 6 are Sunday to Saturday)
    ▒ ▒ +---------- month (1 - 12)
    ▒ +--------------- day of month (1 - 31)
    +-------------------- hour (0 - 23)

    example: orachk -set "AUTORUN_SCHEDULE=8,20 * * 2,5" will schedule runs on tuesday and friday at 8 and 20 hour.

    AUTORUN_FLAGS : orachk flags to use for auto runs.

    example: orachk -set "AUTORUN_INTERVAL=12h;AUTORUN_FLAGS=-profile sysadmin" to run sysadmin profile every 12 hours

    orachk -set "AUTORUN_INTERVAL=2d;AUTORUN_FLAGS=-profile dba" to run dba profile once every 2 days.

    NOTIFICATION_EMAIL : Comma separated list of email addresses used for notifications by daemon if mail server is configured.

    PASSWORD_CHECK_INTERVAL : Interval to verify passwords in daemon mode

    COLLECTION_RETENTION : Purge orachk collection directories and zip files older than specified days.

    unset the parameter
    example: orachk -unset "AUTORUN_SCHEDULE"

    Print the value of parameter

    Pass specific profile.
    List of supported profiles is same as for -profile.

    Pass comma separated collection names(directory or zip files) to merge collections and prepare single report.
    eg:- ./orachk -merge orachk_hostname1_db1_120213_163405.zip,orachk_hostname2_db2_120213_164826.zip

    Pass comma separated filenames containing exalogic guest VM list(applicable for Exalogic only)

    -hybrid [-phy]
    phy :Pass comma separated physical compute nodes(applicable for Exalogic only)
    eg:- ./orachk -hybrid -phy phy_node1,phy_node2

    -profile Pass specific profile.
    List of supported profiles:

    asm asm Checks
    clusterware Oracle clusterware checks
    compute_node Compute Node checks (Exalogic only)
    control_VM Checks only for Control VM(ec1-vm, ovmm, db, pc1, pc2). No cross node checks
    dba dba Checks
    ebs Oracle E-Business Suite checks
    el_extensive Extensive EL checks
    el_lite Exalogic-Lite Checks(Exalogic Only)
    el_rackcompare Data Collection for Exalogic Rack Comparison Tool(Exalogic Only)
    goldengate Oracle GoldenGate checks
    maa Maximum Availability Architecture Checks
    obiee obiee Checks(Exalytics Only)
    storage Oracle Storage Server Checks
    switch Infiniband switch checks
    sysadmin sysadmin checks
    timesten timesten Checks(Exalytics Only)
    virtual_infra OVS, Control VM, NTP-related and stale VNICs check (Exalogic Only)
    zfs ZFS storage appliances checks (Exalogic Only)

    Pass comma separated storage server names to run orachk only on selected storage servers.

    Pass comma separated infiniband switch names to run orachk only on selected infiniband switches.

    Pass comma separated ZFS storage appliance names to run orachk only on selected storage appliances.

    ORAchk Other Useful Options Not Covered Here

  • Using ORAchk Silently
  • ORAchk can be optionally run in “silent” or “non-interactive” mode in order to enable scheduling and automation
    Is required only if customer does not want to use orachk daemon functionality.

  • Using ORAchk Daemon Mode Operation
  • This functionality permit non-interactive (batch or silent mode) execution on a regular interval.

    When running ORAchk in daemon mode, the most recent and next most recent (if any) collection reports are automatically compared. If the mail address is configured a summary will be emailed along with attachments for the reports and the comparison report.

  • Report Comparisons with ORAchk
  • ORAchk has the ability to perform report comparisons between 2 ORAchk reports.
    This allows for trending of Success Factor and Best Practice changes over time, after planned maintenance, etc within a user friendly HTML report.

  • ORAchk in Upgrade Readiness Mode
  • ORAchk can be used to obtain an automated (or above) Upgrade Readiness Assessment.
    The goal of the ORAchk Upgrade Readiness Assessment is to make the process of upgrade planning for Oracle RAC and Oracle Clusterware target versions and above as smooth as possible by automating many of the
    manual pre and post checks detailed in various upgrade related documents.

    Refer MoS Notes for more details:
    ORAchk - Oracle Configuration Audit Tool (Doc ID 1268927.2)

    ORAchk Users Guide
    For details instructions on how to run ORAchk including troubleshooting steps, available options, etc.



    Local/SCAN Listener – Enhancing Security (Oracle Security Alert)

    Recently we discovered  a possible vulnerability on SCAN Listener,  so we opened   SR  and Oracle give us a solution.

    I recommend all apply this security. “As far as I know only the availability can be affected, none concern about data integrity” .

    Thread: How prevent REMOTE LISTENER register on SCAN LISTENER

    Oracle Security Alert for CVE-2012-1675

    This security alert addresses the security issue CVE-2012-1675, a vulnerability in the TNS listener which has been recently disclosed as “TNS Listener Poison Attack” affecting the Oracle Database Server. This vulnerability may be remotely exploitable without authentication, i.e. it may be exploited over a network without the need for a username and password. A remote user can exploit this vulnerability to impact the confidentiality, integrity and availability of systems that do not have recommended solution applied.

    Affected Products and Versions
    Oracle Database 11g Release 2, versions,
    Oracle Database 11g Release 1, version
    Oracle Database 10g Release 2, versions,,


    Recommendations for protecting against this vulnerability can be found at:

    Please note that Oracle has added Oracle Advanced Security SSL/TLS to the Oracle Database Standard Edition license when used with the Real Application Clusters and Oracle has added Oracle Advanced Security SSL/TLS to the Enterprise Edition Real Application Clusters (Oracle RAC) and RAC One Node options so that the directions provided in the Support Notes referenced above can be applied by all Oracle customers without additional cost.

    Note: Please refer to the Oracle licensing documentation available on Oracle.com regarding licensing changes that allow Oracle Advanced Security SSL/TLS to be used with Oracle SE Oracle Real Application Clusters and Oracle Enterprise Edition Real Application Customers (Oracle RAC) and Oracle RAC OneNode Options.

    Due to the threat posed by a successful attack, Oracle strongly recommends that customers apply this Security Alert solution as soon as possible.



    RACcheck – RAC Configuration Audit Tool

    RACcheck is a tool developed by the RAC Assurance development team for use by customers to automate the assessment of RAC systems for known configuration problems and best practices.

    RACcheck is a RAC Configuration Audit tool  designed to audit various important configuration settings within a Real Application Clusters (RAC), Oracle Clusterware (CRS), Automatic Storage Management (ASM) and Grid Infrastructure environment. The tool audits configuration settings within the following categories:

    1. OS kernel parameters
    2. OS packages
    3. Many other OS configuration settings important to RAC.
    4. CRS/Grid Infrastructure
    5. RDBMS
    6. ASM
    7. Database parameters
    8. Many other database configuration settings important to RAC.

    1. RACcheck is NON-INTRUSIVE and does not change anything in the environment, except as detailed below:

    – SSH user equivalence for the RDBMS software owner is assumed to be configured among all the database servers being audited in order for it to execute commands on the remote database server nodes. If the tool determines that this user equivalence is not established it will offer to set it up either temporarily or permanently at the option of the user. If the user chooses to set up SSH user equivalence temporarily then the script will do so for the duration of the execution of the tool but then it will return the system to the state in which it found SSH user equivalence originally. For those wishing to configure SSH user equivalence outside the tool (if not already configured), consult My Oracle Support Note: 372795.1.

    – RACcheck creates a number of small output files into which the data necessary to perform the assessment is collected

    – RACcheck creates and executes some scripts dynamically in order to accomplish some of the data collection

    – RACcheck cleans up after itself any temporary files that are created and not needed as part of the collection.

    2. RACcheck interrogates the system to determine the status of the Oracle stack components (ie., Grid Infrastructure, RDBMS, RAC, etc) and whether they are installed and/or running. Depending upon the status of each component, the tool runs the appropriate collections and audit checks. If due to local environmental configuration the tool is unable to properly determine the needed environmental information please refer to the TROUBLESHOOTING section.

    3. Watchdog daemon – RACcheck automatically runs a daemon in the background to monitor command execution progress. If, for any reason, one of the commands run by the tool should hang or take longer than anticipated, the monitor daemon kills the hung command after a configurable timeout so that main tool execution can progress. If that happens then the collection or command that was hung is skipped and a notation is made in the log. If the default timeout is too short please see the TROUBLESHOOTING section regarding adjustment of the RAT_TIMEOUT, and RAT_ROOT_TIMEOUT parameters.

    4. If RACcheck’s driver files are older than 90 days, the driver files are considered to be “stale” and the script will notify the user of a stale driver file. A new version of the tool and its driver files (kit) must be obtained from MOS Note 1268927.1.

    5. When the RACcheck completes the collection and analysis it produces two reports, summary and detailed. A output .zip file is also produced by RACcheck. This output .zip file can be provided to Oracle Support for further analysis if an SR needs to be logged. The detailed report will contain Benefit/Impact, Risk and Action/Repair information. In many cases it will also reference publicly available documents with additional information about the problem and how to resolve it.

    6. The results of the audit checks can be optionally uploaded into database tables for reporting purposes. See below for more details on this subject.

    7. In some cases customers may want to stage RACcheck on a shared filesystem so that it can be accessed from various systems but be maintained in a single location rather than being copied to each cluster on which it may be used. The default behavior of the tool is to create a subdirectory and its output files in the location where the tool is staged. If that staging area is a read only filesystem or if the user for any reason would like the output to be created elsewhere then there is an environment variable which can be used for that purpose. The RAT_OUTPUT parameter can be set to any valid writable location and the output will be created there.

    Applies to:
    Oracle Server – Enterprise Edition – Version: to – Release: 10.2 to 11.2

    • Linux x86
    • IBM AIX on POWER Systems (64-bit)
    • Oracle Solaris on SPARC (64-bit)
    • Linux x86-64

    To download RAC Check tool use this note on MoS:
    RACcheck – RAC Configuration Audit Tool [ID 1268927.1]

    Example of report output:

    raccheck Report


    Load Balancing and Failover with Oracle 10gR2 RAC

    Oracle Net is a software component that resides on the client and on the Oracle database server. It establishes and maintains the connection between the client application and the server, and exchanges messages between them using industry standard protocols. For the client application and a database to communicate, the client application must specify location details for the database it wants to connect to, and the database must provide some sort of identification or address.On the database server, the Oracle Net Listener, commonly known as the Listener, is a process that listens for client connection requests. The configuration file for the Listener is the listener.ora.The client uses a connect descriptor to specify the database to which to connect. This connect descriptor contains a protocol and a database service name. When a client requests a connection, the Listener on the server receives the request and forwards the connection to the Oracle database. You can define your connect descriptors in the tnsnames.ora file on the client machine, or include them as part of the connection request.When the client connects to the cluster database using a service, you can use the Oracle Net connection load balancing feature to spread user connections across all of the instances that are supporting that service. There are two types of load balancing that you can implement: client-side and server-side load balancing.In an Oracle RAC database, client connections should use both types of connection load balancing. When you create an Oracle RAC database using Oracle Database Configuration Assistant (DBCA), DBCA configures and enables server-side load balancing by default.



    Oracle Real Application Clusters on IBM AIX Best practices in memory tuning and configuring for system stability


    Customers who experience Oracle Real Application Clusters (RAC) node evictions due to excessive AIX kernel paging should carefully review and implement these recommended best practices. Testing and experience have found that memory over commitments may cause scheduling delays for Oracle’s ‘oprocd’ process resulting in node evictions.
    Implementing all of these recommendations will reduce scheduling delays and corresponding oprocd initiated evictions.

    Problem validation

    This paper addresses the best practices for environments experience node evictions caused bycritical processes not being able to get scheduled in a timely fashion on AIX due to memory overcommitment. To validate that node evections are caused by this situation, the followingvalidation steps should be taken.

    Click link below…

    rac_aix_memory_tuning October 17 2011


    Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity

    The purpose of this white paper is to document the setup and configuration of IBM Power Systems™10 Gigabit Ethernet ports.

    The environment used for the tests documented in this paper consist ofthe Oracle Database 11g with Real Application Clusters (RAC) software to configure nodes withprivate network intercommunication in the AIX® operating system environment. Tests were run on aPOWER6™ processor-based Power® 570 server.

    The 10 Gigabit Ethernet cards are an option forthe Power 570.


    This document additionally covers the setup and configuration of AIX EtherChannel for Oracle RACinterconnectivity and the Ethernet switches.

    This document does not cover the installation of the Virtual IO server (VIOS), Logical Partition(LPAR) creation, installation of the AIX operating system and various Oracle RAC components.

    See link below…

    EtherchannelSetup_10GigE_IVE_Power 082510


    Minimum Software Versions and Patches Required to Support Oracle Products on IBM Power Systems

    This is a dynamic document; users should check back for any document updates before installing or updating software. This document contains:

    The latest required software versions and patches for both IBM and Oracle software to enable Oracle products using AIX 6 or AIX 5L on IBM System p servers.

    The latest information about significant interoperability issues and frequently encountered problems.

    Use this Note:

    Minimum Software Versions and Patches Required to Support Oracle Products on IBM Power Systems [ID 282036.1]