Modifying Oracle Clusterware/Restart Binaries

Oracle Database 11.2 introduced a new security features that is lock the GRID_HOME after installation if is required to modify the binaries  GRID_HOME will be necessary to unlock to allow make changes on GRID_HOME .

This procedure must be performed especially when we need to patch GRID_HOME.

Do not force modify Oracle Clusterware/Restart binaries with root  user. This can leave your installation inconsistent.

Let’s do tests:

Upgrading OPatch from  version 11.2.0.1.1 to version 11.2.0.1.5.
The task is quite simple, just rename the directory $GRID_HOME/OPatch to $GRID_HOME/OPatch_old and unpack  new OPatch on  GRID_HOME.
 
[oracle@grid_server1 ~]$ echo $PATH
... :/u01/app/11.2.0/grid/bin:/u01/app/11.2.0/grid/OPatch:/u01/app/11.2.0/grid/jdk/bin...
[oracle@grid_server1 ~]$ echo $ORACLE_HOME
/u01/app/11.2.0/grid
OPatch version
 
[oracle@grid_server1 ~]$ opatch version
Invoking OPatch 11.2.0.1.1

OPatch Version: 11.2.0.1.1
OPatch succeeded.

Perfoming Upgrade of OPatch

As earlier version of Oracle
Unpack patch 6880880

[oracle@grid_server1 patch]$ unzip p6880880_112000_Linux-x86-64.zip
Archive:  p6880880_112000_Linux-x86-64.zip
   creating: OPatch/
   creating: OPatch/crs/
  inflating: OPatch/crs/patch112.pl
  inflating: OPatch/crs/s_crsconfig_defs
  inflating: OPatch/crs/oracss.pm
   creating: OPatch/crs/log/
 extracting: OPatch/crs/log/dummy
  inflating: OPatch/crs/crsconfig_lib.pm
  inflating: OPatch/crs/installPatch.excl
  inflating: OPatch/crs/auto_patch.pl
  inflating: OPatch/crs/s_crsconfig_lib.pm
  inflating: OPatch/crs/crspatch.pm
  inflating: OPatch/crs/crsdelete.pm
   creating: OPatch/jlib/
  inflating: OPatch/jlib/opatch.jar
  inflating: OPatch/jlib/opatchutil.jar
  inflating: OPatch/jlib/opatchext.jar
  inflating: OPatch/jlib/opatchactions.jar
  inflating: OPatch/jlib/opatchfmw.jar
  inflating: OPatch/jlib/opatchprereq.jar
  inflating: OPatch/emdpatch.pl
  inflating: OPatch/opatch.pl
   creating: OPatch/docs/
  inflating: OPatch/docs/Prereq_Users_Guide.txt
  inflating: OPatch/docs/Users_Guide.txt
  inflating: OPatch/docs/FAQ
   creating: OPatch/opatchprereqs/
   creating: OPatch/opatchprereqs/oui/
  inflating: OPatch/opatchprereqs/oui/knowledgesrc.xml
  inflating: OPatch/opatchprereqs/prerequisite.properties
   creating: OPatch/opatchprereqs/opatch/
  inflating: OPatch/opatchprereqs/opatch/opatch_prereq.xml
  inflating: OPatch/opatchprereqs/opatch/rulemap.xml
  inflating: OPatch/opatchprereqs/opatch/runtime_prereq.xml
  inflating: OPatch/opatch.ini
   creating: OPatch/fmw/
  inflating: OPatch/fmw/application.py
  inflating: OPatch/fmw/main_driver.py
  inflating: OPatch/fmw/init_def.py
  inflating: OPatch/fmw/node_manager.py
  inflating: OPatch/fmw/start_stop.py
  inflating: OPatch/fmw/prereq.py
  inflating: OPatch/opatch
  inflating: OPatch/README.txt
   creating: OPatch/ocm/
  inflating: OPatch/ocm/ocm_platforms.txt
   creating: OPatch/ocm/lib/
  inflating: OPatch/ocm/lib/log4j-core.jar
  inflating: OPatch/ocm/lib/http_client.jar
  inflating: OPatch/ocm/lib/emocmclnt.jar
  inflating: OPatch/ocm/lib/regexp.jar
  inflating: OPatch/ocm/lib/osdt_jce.jar
  inflating: OPatch/ocm/lib/jsse.jar
  inflating: OPatch/ocm/lib/osdt_core3.jar
  inflating: OPatch/ocm/lib/xmlparserv2.jar
  inflating: OPatch/ocm/lib/jnet.jar
  inflating: OPatch/ocm/lib/emocmclnt-14.jar
  inflating: OPatch/ocm/lib/emocmcommon.jar
  inflating: OPatch/ocm/lib/jcert.jar
   creating: OPatch/ocm/bin/
  inflating: OPatch/ocm/bin/emocmrsp
 extracting: OPatch/ocm/ocm.zip
  inflating: OPatch/opatch.bat

And moving OPatch directory to $GRID_HOME

[oracle@grid_server1 patch]$ mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch_old
mv: cannot move `/u01/app/11.2.0/grid/OPatch/' to `/u01/app/11.2.0/grid/OPatch_old': Permission denied
[oracle@grid_server1 patch]$ mv OPatch/ $GRID_HOME/OPatch
mv: cannot move `/home/oracle/patch/OPatch/' to `/u01/app/11.2.0/grid/OPatch': Permission denied

So, now we need to unlock GRID_HOME to make the necessary changes.

To unlock GRID_HOME we need execute rootcrs.pl (to a clusterware installation) or roothas.pl (to Grid Standalone/Restart Installation). This files are in $GRID_HOME/crs/install/.

IMPORTANT: Executing rootcrs.pl or roothas.pl with -unlock option the ohasd daemon will be shutdown by rootcrs.pl/roothas.pl. That means that all Oracle services/instances/apps will be shut down on this Node.

With awareness that the environment will be turned off. Let’s do the job.

As root privileges

[oracle@grid_server1 ]$ cd /u01/app/11.2.0/grid/crs/install
[oracle@grid_server1 ]$ sudo perl rootcrs.pl -unlock -crshome /u01/app/11.2.0/grid
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'grid_server1'
CRS-2673: Attempting to stop 'ora.crsd' on 'grid_server1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'grid_server1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.DG_CRS_VOTE.dg' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.db11g.db' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.cvu' on 'grid_server1'
CRS-2677: Stop of 'ora.cvu' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'grid_server1' succeeded
CRS-2673: Attempting to stop 'ora.grid_server1.vip' on 'grid_server1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'grid_server1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'grid_server1'
CRS-2677: Stop of 'ora.scan1.vip' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.grid_server1.vip' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.db11g.db' on 'grid_server1' succeeded
CRS-2673: Attempting to stop 'ora.DG_DATA.dg' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.DG_FRA.dg' on 'grid_server1'
CRS-2677: Stop of 'ora.registry.acfs' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.DG_FRA.dg' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.DG_DATA.dg' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.DG_CRS_VOTE.dg' on 'grid_server1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'grid_server1'
CRS-2677: Stop of 'ora.asm' on 'grid_server1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'grid_server1'
CRS-2677: Stop of 'ora.ons' on 'grid_server1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'grid_server1'
CRS-2677: Stop of 'ora.net1.network' on 'grid_server1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'grid_server1' has completed
CRS-2677: Stop of 'ora.crsd' on 'grid_server1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.evmd' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.asm' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'grid_server1'
CRS-2677: Stop of 'ora.crf' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.asm' on 'grid_server1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'grid_server1'
CRS-2677: Stop of 'ora.evmd' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'grid_server1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'grid_server1'
CRS-2677: Stop of 'ora.cssd' on 'grid_server1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'grid_server1'
CRS-2673: Attempting to stop 'ora.diskmon' on 'grid_server1'
CRS-2677: Stop of 'ora.diskmon' on 'grid_server1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'grid_server1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'grid_server1'
CRS-2677: Stop of 'ora.gpnpd' on 'grid_server1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'grid_server1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /u01/app/11.2.0/grid

Now we can make necessary changes.

[oracle@grid_server1 patch]$ mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch_old
[oracle@grid_server1 patch]$ mv OPatch/ $GRID_HOME/OPatch
[oracle@grid_server1 patch]$  ls -ltr $ORACLE_HOME/ |grep OP*
drwxrwxr-x  8 oracle oinstall  4096 Mar 23 08:24 OPatch
drwxr-xr-x  8 oracle oinstall  4096 Jun 22 17:38 OPatch_old

After finish Upgrade of OPatch we need relock the GRID_HOME and restart the cluster on this node.

To relock the GRID_HOME we need use the rootcrs.pl/roothas.pl with flag -patch. (Not exists the flag -lock)

To see the full manpage for rootcrs.pl/roothas.pl, execute:
perldoc rootcrs.pl
perldoc roothas.pl

[oracle@grid_server1 ]$ cd /u01/app/11.2.0/grid/crs/install
[oracle@grid_server1 ]$ sudo perl rootcrs.pl -patch -crshome /u01/app/11.2.0/grid
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Undefined subroutine &main::read_file called at /u01/app/11.2.0/grid/crs/install/crspatch.pm line 86.

If you get error above, congratulation you have a Oracle bug.
This bug can be easily solved using MOS DOC.ID:
roothas.pl -patch or rootcrs.pl -patch Fails with ‘Undefined subroutine’ [ID 1268390.1]

Let’s solve with this workaround.

  Take a backup of the file /crs/install/crsconfig_lib.pm

# cd /crs/install
# cp crsconfig_lib.pm crsconfig_lib.pm.bak

Make the following change in that file crsconfig_lib.pm

From
     my @exp_func = qw(check_CRSConfig validate_olrconfig validateOCR
To
     my @exp_func = qw(check_CRSConfig validate_olrconfig validateOCR read_file

Execute relock:

[oracle@grid_server1 ]$ cd /u01/app/11.2.0/grid/crs/install
[oracle@grid_server1 ]$ sudo perl rootcrs.pl -patch -crshome /u01/app/11.2.0/grid
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
ACFS-9200: Supported
CRS-4123: Oracle High Availability Services has been started.

Testing if all services are online and GRID_HOME is LOCKED.

[oracle@grid_server1 ~]$ crsctl status resource -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DG_CRS_VOTE.dg
               ONLINE  ONLINE       grid_server1
               ONLINE  ONLINE       grid_server2
ora.DG_DATA.dg
               ONLINE  ONLINE       grid_server1
               ONLINE  ONLINE       grid_server2
ora.DG_FRA.dg
               ONLINE  ONLINE       grid_server1
               ONLINE  ONLINE       grid_server2
ora.LISTENER.lsnr
               ONLINE  ONLINE       grid_server1
               ONLINE  ONLINE       grid_server2
ora.asm
               ONLINE  ONLINE       grid_server1
               ONLINE  ONLINE       grid_server2
ora.gsd
               OFFLINE OFFLINE      grid_server1
               OFFLINE OFFLINE      grid_server2
ora.net1.network
               ONLINE  ONLINE       grid_server1
               ONLINE  ONLINE       grid_server2
ora.ons
               ONLINE  ONLINE       grid_server1
               ONLINE  ONLINE       grid_server2
ora.registry.acfs
               ONLINE  ONLINE       grid_server1
               ONLINE  ONLINE       grid_server2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       grid_server1
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       grid_server2
ora.grid_server1.vip
      1        ONLINE  ONLINE       grid_server1
ora.cvu
      1        ONLINE  ONLINE       grid_server2
ora.db11g.db
      1        ONLINE  ONLINE       grid_server1                 Open
      2        ONLINE  ONLINE       grid_server2                  Open
ora.grid_server2.vip
      1        ONLINE  ONLINE       grid_server2
ora.oc4j
      1        ONLINE  ONLINE       grid_server1
ora.scan1.vip
      1        ONLINE  ONLINE       grid_server1
ora.scan2.vip
      1        ONLINE  ONLINE       grid_server2

 

[oracle@grid_server1 ~]$ mv /u01/app/11.2.0/grid/OPatch /u01/app/11.2.0/grid/OPatch_tst
mv: cannot move `/u01/app/11.2.0/grid/OPatch' to `/u01/app/11.2.0/grid/OPatch_tst': Permission denied

[oracle@grid_server1 ~]$ opatch version
Invoking OPatch 11.2.0.1.5

OPatch Version: 11.2.0.1.5
OPatch succeeded.

Enjoy

Reference:
Modifying Oracle Clusterware Binaries After Installation
http://download.oracle.com/docs/cd/E11882_01/install.112/e17210/postinst.htm#CWAIX311

Patch 11gR2 Grid Infrastructure Standalone Server [ID 1089476.1]
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=ANNOUNCEMENT&id=1089476.1

Advertisements

7 Comments on “Modifying Oracle Clusterware/Restart Binaries”

  1. Wissem says:

    Hi Levi,
    I have successfully applied the PSU Patch 12311357 – 11.2.0.2.2 GI Patch Set Update on both GI and OH on RAC 11gR2 4 nodes on redhat Linux 5.5, But, and don’t remember I had to unlock the GRID_HOME.
    is this correct?

    Thank you

    Like

    • Levi Pereira says:

      Hi,
      I liked the question.
      In the case of PSU patch is correct. You don’t need unlock/relock GRID_HOME manually.
      The Patch 12311357 is PSU. Patch Set Updates (PSUs) are proactive cumulative patches containing recommended bug fixes that are released on a regular and predictable schedule. (like Advanced Patch)
      In releases prior to Oracle 11.2 PSU was applied with the Installation user (e.g oracle) on CRS_HOME (ie: non-root user), but in Oracle 11.2 the PSU must be applied with root privileges.
      Oracle Says:
      The utility (opatch) must be executed by an operating system (OS) user with root privileges (usually the user root), and it must be executed on each node in the cluster if the GI home or Oracle RAC database home is in Non-shared storage.

      So, opatch with root privileges unlock and relock the GRID_HOME automatically.

      But in case of One-off Patches you must unlock the GRID_HOME.

      e.g : See this patch : 10420872 11.2.0.2 GRID INFRASTRUCTURE ROOT.SH FAILS

      On step 4. (with root user)
      4. Prior to applying this part of the fix, you must invoke this script as root to unlock protected files.

      On step 6: (with Instalation owner)
      As the Oracle Clusterware (CRS) software owner, from the directory where the patch was unzipped;
      % opatch napply -local -oh -id 10420872

      Cheers,
      Levi Pereira

      Like

  2. Wissem says:

    Thank you Levi, I get it know 🙂

    I have another quick question, I am trying to apply the Patch 12311357 on my standalone grid infrastructure and single instance 11.2.0.2.

    But the thing is when trying to apply the PSU, it says:

    Cann’t open /a01/app/oracle/product/11.2.
    0/db_1/css/admin/init.cssd at /a01/app/oracle/product/11.2.0/db_1/OPatch/crs/auto_patch.pl line 3161.

    It seems like the PSU considers that is a cluster database. In all the readme document it doesn’t mention single instance database.

    3 weeks ago, I successfully applied it in RAC 4 nodes, without issues.

    I wonder if you have tried it on a single instance?

    Thank you Levi,
    Wissem

    Like

    • Levi Pereira says:

      Hi Wissem,

      Yes … but I don’t get this error … the full process was successful. I’ll do some tests and see you reproduce the error.

      I suggest you try an answer in Oracle Forums.

      Regards,
      Levi Pereira

      Like

  3. Wissem says:

    Hi levi,
    I successfully applied it, here my experience 🙂

    http://wp.me/p1hlba-xD

    thanks,
    Wissem

    Like

  4. […] server (Oracle Restart) can be under Oracle base. Re: OFA & Grid Infrastructure home Modifying Oracle Clusterware/Restart Binaries Regards, Reply With […]

    Like

  5. […] Modifying Oracle Clusterware/Restart Binaries | Levi … – Jul 01, 2011 · Oracle Database 11.2 introduced a new security features that is lock the GRID_HOME after installation if is required to modify the binaries GRID …… […]

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s