PRCR-1079 CRS-5017 ORA-01017 – DBCA Fails to create a Database in Oracle Restart Environment 12c.

The Oracle Grid Infrastructure software owner (typically, grid) must be a member of the OSDBA group. Membership in the OSDBA group enables access to the files managed by Oracle ASM. If you have a separate OSDBA group for Oracle ASM, then the Oracle Restart software owner must be a member of the OSDBA group for each database and the OSDBA group for Oracle ASM.

http://docs.oracle.com/database/121/LADBI/usr_grps.htm#BABFECII

Issue:

If grid user is not a member of dba os group, then DBCA will fails at end of database creation and DBCA will perform rollback of database creation (database will not be created).

The error below will be raised:

srvctl start database -d prdcdb
PRCR-1079 : Failed to start resource ora.prdcdb.db
ORA-01017: invalid username/password; logon denied
CRS-5017: The resource action "ora.prdcdb.db start" encountered the following
error: ORA-01017: invalid username/password; logon denied
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/db-oracle/crs/
trace/ohasd_oraagent_grid.trc".

 

Solution:

To fix the issue you must add  user “grid” to OSDBA group “dba” and relink RDBMS Binaries.

# id grid
uid=205(grid) gid=202(oinstall) groups=208(asmdba),209(asmadmin),210(asmoper)
# usermod -a -G dba grid (linux)
# id grid
uid=205(grid) gid=202(oinstall) groups=203(dba),208(asmdba),209(asmadmin),
210(asmoper)


# su - oracle
oracle@db-oracle:/home/oracle> export AIXTHREAD_SCOPE=S
oracle@db-oracle:/home/oracle> export ORACLE_BASE=/u01/app/oracle
oracle@db-oracle:/home/oracle> export ORACLE_HOME=$ORACLE_BASE/product/12.1.0/
dbhome_12102
oracle@db-oracle:/home/oracle> export PATH=$ORACLE_HOME/bin:$PATH

oracle@db-oracle:/home/oracle> relink all
writing relink log to: /u01/app/oracle/product/12.1.0/dbhome_12102/
install/relink.log


If there are no errors reported in $ORACLE_HOME/install/relink.log, then retry to create the database with dbca.

 

 

 


Starting with Oracle In-Memory (aka IM) new Feature


I recommend that DBA be aware of this feature, because this feature will really bring a HUGE difference in performance matter to our databases.

Note: This feature is available starting with Oracle Database 12c Release 1 (12.1.0.2).

Good to Know

  • Is a Option of Oracle Enterprise Edition (12.1.0.2 and above). - Additional license required
  • It is Not in Memory Database - it's an accelerator to our current database
  • It is Not Column Store Database - it allows keeping some of our data in column store which is non-persistent
  • It has nothing to do with Oracle Times-Ten or Oracle Coherence
  • No additional hardware requirement, except additional OS RAM available

Price: Buy it

 

I'm posting useful videos links from Oracle Learning Library.

 

White paper: When to Use Oracle Database In-Memory (PDF)

White paper: Oracle Database In-Memory Advisor Best Practices (PDF)

Blog Official of In-Memory Feature

 


How to move/migrate OCR, Voting Disk, ASM (SPFILE and PWDFILE) and MGMTDB to a different Diskgroup on ASM 12.1.0.1

_______________________________________
Last Update (29/09/2014):
Oracle released the note: How to Move GI Management Repository to Different Shared Storage (Diskgroup, CFS or NFS etc) (Doc ID 1589394.1)

The procedure on NOTE (Doc ID 1589394.1) does not "MOVE" but RECREATE Management Repository.
The procedure below on this post MOVE (migrating all data) from a Diskgroup to another Diskgroup.

_______________________________________

This post is only a recommendation about how to configure Clusterware Files and MGMTDB on ASM using Grid Infrastructure 12.1.0.1.

Migrating MGMTDB to a different Diskgroup is not documented until 25/09/2013. Oracle will release a note to make this procedure soon.

When we install/configure a new fresh Oracle GI for a Cluster the cluster files and MGMTDB is stored on first Diskgroup created at moment of installation.

To easy management and increase the avability of Cluster is recommended setup small diskgroup to store the cluster files.

ASM Spfile stuck on Diskgroup : ORA-15027: active use of diskgroup precludes its dismount (With no database clients connected) [ID 1082876.1]

The cluster files are:

Oracle Cluster Registry (OCR)

The Oracle RAC configuration information repository that manages information about the cluster node list and instance-to-node mapping information. The OCR also manages information about Oracle Clusterware resource profiles for customized applications.

The OCR is stored in a Diskgroup similar as Database Files are stored. The extents are spread across all disk of  diskgroup and the redundancy (which is at the extent level) is based on the redundancy of the disk group. We can store only one OCR per Diskgroup.

Voting Disk

Voting Disk aka Voting files: Is a file that manages information about node membership.

The voting disks are placed directly on ASMDISK/Luns. Oracle Clusterware will store the votedisk on the disk within a disk group that holds the Voting Files. Oracle Clusterware does not rely on ASM to access the Voting Files, that’s  means wich Oracle Clusterware does not need of Diskgroup to read and write on ASMDISK.
You cannot find/list Voting files using SQLPLUS(v$asm_diskgroup,v$asm_files,v$asm_alias), ASCMD utility or ASMCA gui.
You only know if exist a voting files in a ASMDISK (v$asm_disk using column VOTING_FILE). So, voting files not depend of Diskgroup to be accessed, does not mean that, we don’t need the diskgroup, diskgroup and voting file are linked by their settings and redundancy.

We cannot place Voting Disk in multiples Diskgroup. Is supported use only one Diskgroup to store Voting Disks.

ASM Spfile

This files is a parameter file of ASM Instance.

The Spfile is stored as OCR and Database files.

ASM Password File

New feature of ASM 12.1 allow store Database or ASM Password File on ASM Diskgroup.

An individual password file for Oracle Database or Oracle ASM can reside on a designated Oracle ASM disk group. Having the password files reside on a single location accessible across the cluster reduces maintenance costs and situations where passwords become out of sync.

The COMPATIBLE.ASM disk group attribute must be set to at least 12.1 for the disk group where the password is to be located.

Database MGMTDB

New feature of GI 12.1 Management Repository.

Management Repository is a single instance database that's managed by Oracle Clusterware in 12c. If the option is selected during GI installation, the database will be configured and managed by GI. As it's a single instance database, it will be up and running on one node in the cluster; as it's managed by GI, in case the hosting node is down, the database will be automatically failed over to other node.

Management database by default uses the same shared storage as OCR/Voting File.

There is no procedure in documentation 12.1 or Metalink Notes to change location of MGMTDB. (25/09/2013)

What are my recommendation:

Create two diskgroup:

+OCR_VOTE

Even using RAID at Storage Layer create a small diskgroup with 3 ASMDISK  using Normal Redundancy. This Diskgroup will store Voting Disk, OCR, ASM Spfile,  ASM Password File.

The reason to create this Diskgroup with Normal Redundancy is provide availability  in case of loss of one voting disk or loss of one ASM Disk, caused by human errors or storage failure  or host failure (controller).

+MGMTDB

This Diskgroup can be created using External Redudancy if you are using RAID at Storage Layer.

This Diskgroup  will store MGMTDB and OCR Mirror.

Important:  To perform the changes below all nodes in the cluster must be active. 

Let's Play:

Checking if nodes are actives.

	
[grid@node12c02 ~]$ olsnodes -s
node12c01        	Active
node12c02 			Active

Use OCRCHECK to find where OCR file is stored.

	
[grid@node12c02 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1548
         Available space (kbytes) :     408020
         ID                       : 1037914462
         Device/File Name         :   +CRS_TMP
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

Use crsctl to find where Voting Disk is stored.

	
[grid@node12c02 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   0d2d3b5d705a4fefbf8824c99251ac5a (/dev/oracleasm/disks/OCR_TEMP) [CRS_TMP]
Located 1 voting disk(s).

Use asmcmd to find where ASM SPFILE and ASM PWD is stored

	
[grid@node12c02 ~]$ asmcmd spget
+CRS_TMP/crs12c/ASMPARAMETERFILE/registry.253.826898607

[grid@node12c02 ~]$ asmcmd pwget --asm
+CRS_TMP/crs12c/orapwASM

Getting info about Voting Disk on ASM.

Connect to ASM using as sysdba role

SQL>
SET LINESIZE 150
COL PATH FOR A30
COL NAME FOR A20
COL HEADER_STATUS FOR A20
COL FAILGROUP FOR A20
COL FAILGROUP_TYPE FOR A20
COL VOTING_FILE FOR A20
SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE
FROM V$ASM_DISK
WHERE GROUP_NUMBER = ( SELECT GROUP_NUMBER
FROM V$ASM_DISKGROUP
WHERE NAME='CRS_TMP');

			 
NAME                     PATH                           HEADER_STATUS        FAILGROUP            FAILGROUP_TYPE       VOTING_FILE
------------------------ ------------------------------ -------------------- -------------------- -------------------- --------------------
CRS_TMP_0000             /dev/oracleasm/disks/OCR_TEMP  MEMBER               CRS_TMP_0000         REGULAR              Y

Get name of cluster.

	
[grid@node12c02 ~]$ olsnodes -c
crs12c

set linesize 100
col FILES_OF_CLUSTER for a60
select concat('+'||gname, sys_connect_by_path(aname, '/')) FILES_OF_CLUSTER
from ( select b.name gname, a.parent_index pindex, a.name aname,
a.reference_index rindex , a.system_created, a.alias_directory,
c.type file_type
from v$asm_alias a, v$asm_diskgroup b, v$asm_file c
where a.group_number = b.group_number
and a.group_number = c.group_number(+)
and a.file_number = c.file_number(+)
and a.file_incarnation = c.incarnation(+)
) WHERE file_type in ( 'ASMPARAMETERFILE','PASSWORD','OCRFILE')
start with (mod(pindex, power(2, 24))) = 0
and rindex in
( select a.reference_index
from v$asm_alias a, v$asm_diskgroup b
where a.group_number = b.group_number
and (mod(a.parent_index, power(2, 24))) = 0
and a.name = LOWER('&CLUSTERNAME')
)
connect by prior rindex = pindex;

	
Enter value for clustername: crs12c
old  17:                         and a.name = LOWER('&CLUSTERNAME')
new  17:                         and a.name = LOWER('crs12c')

FILES_OF_CLUSTER
------------------------------------------------------------
+CRS_TMP/crs12c/ASMPARAMETERFILE/REGISTRY.253.826898607
+CRS_TMP/crs12c/OCRFILE/REGISTRY.255.826898619
+CRS_TMP/crs12c/orapwasm

Creating Diskgroup OCR_VOTE

Creating Diskgroup OCR_VOTE each disk must be in different failgroup. I don’t add QUORUM failgroup because theses luns are on Storage. Use QUORUM failgroup when you are placing disk out of your environment. (e.g use NFS file-disk to quorum purpouses), because this disks cannot contain data.

List ASM Disk
SQL>
col path for a30
col name for a20
col header_status for a20
select path,name,header_status from v$asm_disk
where path like '/dev/asm%';

	 
 PATH                           NAME                 HEADER_STATUS
------------------------------ -------------------- --------------------
/dev/asm-ocr_vote3                                  CANDIDATE
/dev/asm-ocr_vote1                                  CANDIDATE
/dev/asm-ocr_vote2                                  CANDIDATE          
	
SQL> CREATE DISKGROUP OCR_VOTE NORMAL REDUNDANCY
     FAILGROUP controller01 DISK '/dev/asm-ocr_vote1'
     FAILGROUP controller02 DISK '/dev/asm-ocr_vote2'
     FAILGROUP controller03 DISK '/dev/asm-ocr_vote3'
     ATTRIBUTE 
	 'au_size'='1M',
	 'compatible.asm' = '12.1';

Diskgroup created.

Mount Diskgroup OCR_VOTE on others Nodes

	
SQL>  ! srvctl start diskgroup -g ocr_vote -n node12c01

Check status of Diskgroup OCR_VOTE in all nodes.
The Diskgroup OCR_VOTE must be mounted in all hub nodes.

	
SQL> ! srvctl status diskgroup -g ocr_vote
Disk Group ocr_vote is running on node12c01,node12c02

Moving Voting Disk to Diskgroup OCR_VOTE

As grid user issue:

	
[grid@node12c01 ~]$  crsctl replace votedisk +OCR_VOTE
Successful addition of voting disk be10b93cca734f65bfa6b89cc574d485.
Successful addition of voting disk 46a9b6154c184f89bf6a503846ea3aa3.
Successful addition of voting disk 6bc41f75098d4f5dbf97783770760445.
Successful deletion of voting disk 0d2d3b5d705a4fefbf8824c99251ac5a.
Successfully replaced voting disk group with +OCR_VOTE.
CRS-4266: Voting file(s) successfully replaced

Query CSS Votedisk in all nodes

	
[grid@node12c01 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   be10b93cca734f65bfa6b89cc574d485 (/dev/asm-ocr_vote1) [OCR_VOTE]
 2. ONLINE   46a9b6154c184f89bf6a503846ea3aa3 (/dev/asm-ocr_vote2) [OCR_VOTE]
 3. ONLINE   6bc41f75098d4f5dbf97783770760445 (/dev/asm-ocr_vote3) [OCR_VOTE]
Located 3 voting disk(s).

[grid@node12c02 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   be10b93cca734f65bfa6b89cc574d485 (/dev/asm-ocr_vote1) [OCR_VOTE]
 2. ONLINE   46a9b6154c184f89bf6a503846ea3aa3 (/dev/asm-ocr_vote2) [OCR_VOTE]
 3. ONLINE   6bc41f75098d4f5dbf97783770760445 (/dev/asm-ocr_vote3) [OCR_VOTE]
Located 3 voting disk(s).

Query ASMDISK on ASM
SQL >
SET LINESIZE 150
COL PATH FOR A30
COL NAME FOR A10
COL HEADER_STATUS FOR A20
COL FAILGROUP FOR A20
COL FAILGROUP_TYPE FOR A20
COL VOTING_FILE FOR A20
SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE
FROM V$ASM_DISK
WHERE GROUP_NUMBER = ( SELECT GROUP_NUMBER
FROM V$ASM_DISKGROUP
WHERE NAME='OCR_VOTE');

	
NAME                     PATH                           HEADER_STATUS        FAILGROUP            FAILGROUP_TYPE       VOTING_FILE
------------------------ ------------------------------ -------------------- -------------------- -------------------- --------------------
OCR_VOTE_0002            /dev/asm-ocr_vote3             MEMBER               CONTROLLER03         REGULAR              Y
OCR_VOTE_0001            /dev/asm-ocr_vote2             MEMBER               CONTROLLER02         REGULAR              Y
OCR_VOTE_0000            /dev/asm-ocr_vote1             MEMBER               CONTROLLER01         REGULAR              Y

The Voting Disk was moved successful.

Moving OCR to Diskgroup OCR_VOTE

As root user issue:

	
[root@node12c02 ~]# /u01/app/12.1.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1568
         Available space (kbytes) :     408000
         ID                       : 1037914462
         Device/File Name         :   +CRS_TMP
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

Add OCR on OCR_VOTE

	
[root@node12c02 ~]# /u01/app/12.1.0/grid/bin/ocrconfig -add +OCR_VOTE	

Now we must see two OCR one in each Diskgroup

	
[root@node12c02 ~]# /u01/app/12.1.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1568
         Available space (kbytes) :     408000
         ID                       : 1037914462
         Device/File Name         :   +CRS_TMP
                                    Device/File integrity check succeeded
         Device/File Name         :  +OCR_VOTE
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

Delete OCR from Diskgroup CRS_TMP

	
[root@node12c02 ~]# /u01/app/12.1.0/grid/bin/ocrconfig -delete +CRS_TMP

Now we must see only one OCR on Diskgroup OCR_VOTE.

	
[root@node12c02 ~]# /u01/app/12.1.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1568
         Available space (kbytes) :     408000
         ID                       : 1037914462
         Device/File Name         :  +OCR_VOTE
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

Check with OCRCHECK on others nodes.

	
[root@node12c01 ~]#  /u01/app/12.1.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1568
         Available space (kbytes) :     408000
         ID                       : 1037914462
         Device/File Name         :  +OCR_VOTE
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

The OCR was moved successful.

Moving ASM SPFILE
You will get the error which the file is still being used, but the file is copied to the file system and the profile is updated.

	
[grid@node12c01 ~]$ asmcmd spmove '+CRS_TMP/crs12c/ASMPARAMETERFILE/registry.253.826898607' '+OCR_VOTE/crs12c/spfileASM.ora'
ORA-15032: not all alterations performed
ORA-15028: ASM file '+CRS_TMP/crs12c/ASMPARAMETERFILE/registry.253.826898607' not dropped; currently being accessed (DBD ERROR: OCIStmtExecute)

Checking if file was copied and profile updated

	
[grid@node12c01 ~]$ asmcmd spget
+OCR_VOTE/crs12c/spfileASM.ora

Moving ASM Password File

	
[grid@node12c01 ~]$ asmcmd pwmove --asm +CRS_TMP/orapwASM +OCR_VOTE/crs12c/orapwASM
moving +CRS_TMP/orapwASM -> +OCR_VOTE/crs12c/orapwASM

Checking if file was moved and profile updated

	
[grid@node12c01 ~]$ asmcmd pwget --asm
+OCR_VOTE/crs12c/orapwASM

[grid@node12c01 ~]$ srvctl config asm
ASM home: /u01/app/12.1.0/grid
Password file: +OCR_VOTE/crs12c/orapwASM
ASM listener: LISTENER

After these changes restart cluster to release ASM SPFILE.
You can restart one node at time if you wish.

You DON'T need stop whole cluster to release ASM SPFILE.

As my cluster is a test cluster I will stop cluster on all nodes.

	
[root@node12c01 ~]# /u01/app/12.1.0/grid/bin/crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'node12c01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node12c01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'node12c01'
CRS-2673: Attempting to stop 'ora.CRS_TMP.dg' on 'node12c01'
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'node12c01'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'node12c01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'node12c01'
CRS-2677: Stop of 'ora.scan1.vip' on 'node12c01' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'node12c02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node12c02'
CRS-2673: Attempting to stop 'ora.cvu' on 'node12c02'
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'node12c02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'node12c02'
CRS-2673: Attempting to stop 'ora.node12c02.vip' on 'node12c02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node12c02'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'node12c02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node12c02'
CRS-2677: Stop of 'ora.cvu' on 'node12c02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'node12c02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'node12c02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node12c02' succeeded
CRS-2677: Stop of 'ora.node12c02.vip' on 'node12c02' succeeded
CRS-2673: Attempting to stop 'ora.node12c01.vip' on 'node12c02'
CRS-2677: Stop of 'ora.scan2.vip' on 'node12c02' succeeded
CRS-2677: Stop of 'ora.node12c01.vip' on 'node12c02' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'node12c02' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'node12c02'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'node12c02' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'node12c01' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'node12c02' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'node12c02' succeeded
CRS-2677: Stop of 'ora.CRS_TMP.dg' on 'node12c01' succeeded
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'node12c01'
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'node12c01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node12c01'
CRS-2677: Stop of 'ora.asm' on 'node12c01' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'node12c01'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'node12c01' succeeded
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'node12c02'
CRS-2673: Attempting to stop 'ora.CRS_TMP.dg' on 'node12c02'
CRS-2673: Attempting to stop 'ora.ons' on 'node12c01'
CRS-2677: Stop of 'ora.ons' on 'node12c01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node12c01'
CRS-2677: Stop of 'ora.net1.network' on 'node12c01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node12c01' has completed
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'node12c02' succeeded
CRS-2677: Stop of 'ora.CRS_TMP.dg' on 'node12c02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node12c02'
CRS-2677: Stop of 'ora.asm' on 'node12c02' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'node12c02'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'node12c02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node12c02'
CRS-2677: Stop of 'ora.ons' on 'node12c02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node12c02'
CRS-2677: Stop of 'ora.net1.network' on 'node12c02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node12c02' has completed
CRS-2677: Stop of 'ora.crsd' on 'node12c02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'node12c02'
CRS-2673: Attempting to stop 'ora.evmd' on 'node12c02'
CRS-2673: Attempting to stop 'ora.storage' on 'node12c02'
CRS-2677: Stop of 'ora.storage' on 'node12c02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node12c02'
CRS-2677: Stop of 'ora.crsd' on 'node12c01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'node12c01'
CRS-2673: Attempting to stop 'ora.evmd' on 'node12c01'
CRS-2673: Attempting to stop 'ora.storage' on 'node12c01'
CRS-2677: Stop of 'ora.storage' on 'node12c01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node12c01'
CRS-2677: Stop of 'ora.evmd' on 'node12c02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node12c01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node12c02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node12c01' succeeded
CRS-2677: Stop of 'ora.asm' on 'node12c01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'node12c01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'node12c01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node12c01'
CRS-2677: Stop of 'ora.cssd' on 'node12c01' succeeded
CRS-2677: Stop of 'ora.asm' on 'node12c02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'node12c02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'node12c02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node12c02'
CRS-2677: Stop of 'ora.cssd' on 'node12c02' succeeded

Start cluster on all nodes

	
[root@node12c01 ~]# /u01/app/12.1.0/grid/bin/crsctl start cluster -all
CRS-2672: Attempting to start 'ora.evmd' on 'node12c01'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node12c01'
CRS-2672: Attempting to start 'ora.evmd' on 'node12c02'
CRS-2676: Start of 'ora.cssdmonitor' on 'node12c01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node12c01'
CRS-2672: Attempting to start 'ora.diskmon' on 'node12c01'
CRS-2676: Start of 'ora.diskmon' on 'node12c01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node12c02'
CRS-2676: Start of 'ora.evmd' on 'node12c01' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'node12c02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node12c02'
CRS-2672: Attempting to start 'ora.diskmon' on 'node12c02'
CRS-2676: Start of 'ora.evmd' on 'node12c02' succeeded
CRS-2676: Start of 'ora.diskmon' on 'node12c02' succeeded
CRS-2676: Start of 'ora.cssd' on 'node12c02' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'node12c02'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'node12c02'
CRS-2676: Start of 'ora.cssd' on 'node12c01' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'node12c01'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'node12c01'
CRS-2676: Start of 'ora.ctssd' on 'node12c02' succeeded
CRS-2676: Start of 'ora.ctssd' on 'node12c01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'node12c02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'node12c02'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'node12c01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'node12c01'
CRS-2676: Start of 'ora.asm' on 'node12c01' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'node12c01'
CRS-2676: Start of 'ora.asm' on 'node12c02' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'node12c02'
CRS-2676: Start of 'ora.storage' on 'node12c02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'node12c02'
CRS-2676: Start of 'ora.crsd' on 'node12c02' succeeded
CRS-2676: Start of 'ora.storage' on 'node12c01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'node12c01'
CRS-2676: Start of 'ora.crsd' on 'node12c01' succeeded

Migrate MGMTDB to other Diskgroup.

MGMTDB is a "common" Oracle RDBMS (exclusive to CHM). We can perform administrative task like in any other Oracle RDBMS.
So, to migrate MGMTDB to other Diskgroup I will use RMAN.

Creating Diskgroup MGMTDB

	
SQL> CREATE DISKGROUP MGMTDB EXTERNAL REDUNDANCY
     DISK '/dev/asm-mgmtdb1'
     ATTRIBUTE 
	 'au_size'='1M',
	 'compatible.asm' = '12.1',
	 'compatible.rdbms' = '12.1';			 
			 
Diskgroup created.

Starting MGMTDB diskgroup on others node and checking status

	
SQL> ! srvctl start diskgroup -g MGMTDB -n node12c02

SQL> ! srvctl status diskgroup -g MGMTDB
Disk Group MGMTDB is running on node12c01,node12c02

To manage MGMTDB you must use srvctl utility.
srvctl start/stop/config/modify... mgmtdb

Getting current configuration of MGMTDB

	

[grid@node12c01 ~]$ srvctl config mgmtdb
Database unique name: _mgmtdb
Database name:
Oracle home: /u01/app/12.1.0/grid
Oracle user: grid
Spfile: +CRS_TMP/_mgmtdb/spfile-MGMTDB.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Database instance: -MGMTDB
Type: Management

Check where MGMTDB is running.

	
[grid@node12c01 ~]$ srvctl status mgmtdb
Database is enabled
Instance -MGMTDB is running on node node12c02

You must migrate MGMTDB using host(node) where MGMTDB is running.

Let's start.
Stop MGMT Listener and MGMTDB

	
[grid@node12c02 ~]$ srvctl stop mgmtdb

[grid@node12c02 ~]$ srvctl stop mgmtlsnr

Precaution: I will create a full backup of the database MGMTDB before start migration.

	
[grid@node12c02 ~]$ mkdir -p /u01/app/grid/backup_mgmtdb

[grid@node12c02 ~]$ export ORACLE_SID=-MGMTDB
[grid@node12c02 ~]$ rman target /

Recovery Manager: Release 12.1.0.1.0 - Production on Tue Sep 24 19:30:56 2013

Copyright (c) 1982, 2013, Oracle and/or its affiliates.  All rights reserved.

connected to target database (not started)

RMAN> startup mount

connected to target database (not started)
Oracle instance started
database mounted

Total System Global Area     521936896 bytes

Fixed Size                     2290264 bytes
Variable Size                289410472 bytes
Database Buffers             222298112 bytes
Redo Buffers                   7938048 bytes


RMAN>  backup database format '/u01/app/grid/backup_mgmtdb/rman_mgmtdb_%U' tag='bk_db_move_dg';

Starting backup at 24-SEP-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=6 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00005 name=+CRS_TMP/_MGMTDB/DATAFILE/sysmgmtdata.260.826899881
input datafile file number=00001 name=+CRS_TMP/_MGMTDB/DATAFILE/system.259.826899845
input datafile file number=00002 name=+CRS_TMP/_MGMTDB/DATAFILE/sysaux.258.826899821
input datafile file number=00004 name=+CRS_TMP/_MGMTDB/DATAFILE/sysgridhomedata.261.826899917
input datafile file number=00003 name=+CRS_TMP/_MGMTDB/DATAFILE/undotbs1.257.826899817
channel ORA_DISK_1: starting piece 1 at 24-SEP-13
channel ORA_DISK_1: finished piece 1 at 24-SEP-13
piece handle=/u01/app/grid/backup_mgmtdb/rman_mgmtdb_03okm63j_1_1 tag=BK_DB_MOVE_DG comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:45
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 24-SEP-13
channel ORA_DISK_1: finished piece 1 at 24-SEP-13
piece handle=/u01/app/grid/backup_mgmtdb/rman_mgmtdb_04okm650_1_1 tag=BK_DB_MOVE_DG comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 24-SEP-13

Migrating SPFILE to Diskgroup +MGMTDB.
RMAN will restore SPFILE and update OCR to a new SPFILE Location.
Database _MGMTDB MUST be mounted or opened.

	

RMAN> restore spfile to "+MGMTDB";

Starting restore at 24-SEP-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=355 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: restoring SPFILE
output file name=+MGMTDB
channel ORA_DISK_1: reading from backup piece /u01/app/grid/backup_mgmtdb/rman_mgmtdb_02oltbpe_1_1
channel ORA_DISK_1: piece handle=/u01/app/grid/backup_mgmtdb/rman_mgmtdb_02oltbpe_1_1 tag=BK_DB_MOVE_DG
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
Finished restore at 24-SEP-13

[grid@node12c01 ~]$ srvctl config mgmtdb |grep Spfile
Spfile: +MGMTDB/_MGMTDB/PARAMETERFILE/spfile.256.828288969

RMAN> shutdown immediate;
database dismounted
Oracle instance shut down

Migrating Controlfile to Diskgroup MGMTDB
P.S: Before do this change, check what spfile you are updating.

	

[grid@node12c02 ~]$ sqlplus / as sysdba
SQL> startup nomount;
ORACLE instance started.

Total System Global Area  521936896 bytes
Fixed Size                  2290264 bytes
Variable Size             285216168 bytes
Database Buffers          226492416 bytes
Redo Buffers                7938048 bytes

SQL> show parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +MGMTDB/_MGMTDB/PARAMETERFILE/spfile.256.828288969


SQL> show parameter control_file

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
control_files                        string      +CRS_TMP/_MGMTDB/CONTROLFILE/current.262.826899953

SQL> alter system set control_files='+MGMTDB' scope=spfile ;

System altered.

SQL> shutdown immediate;
ORA-01507: database not mounted


ORACLE instance shut down.
SQL> exit
[grid@node12c02 ~]$ rman target /

Recovery Manager: Release 12.1.0.1.0 - Production on Tue Sep 24 19:45:37 2013

Copyright (c) 1982, 2013, Oracle and/or its affiliates.  All rights reserved.

connected to target database (not started)

RMAN> startup nomount

Oracle instance started

Total System Global Area     521936896 bytes

Fixed Size                     2290264 bytes
Variable Size                289410472 bytes
Database Buffers             222298112 bytes
Redo Buffers                   7938048 bytes

RMAN> RESTORE CONTROLFILE FROM '+CRS_TMP/_MGMTDB/CONTROLFILE/current.262.826899953';

Starting restore at 24-SEP-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=124 device type=DISK

channel ORA_DISK_1: copied control file copy
output file name=+MGMTDB/_MGMTDB/CONTROLFILE/current.263.827005571
Finished restore at 24-SEP-13

RMAN> startup mount

database is already started
database mounted
released channel: ORA_DISK_1

Controlfile was moved successful.

Migrating Database Files to Diskgroup MGMTDB

	
RMAN>  BACKUP AS COPY DEVICE TYPE DISK DATABASE FORMAT '+MGMTDB';

Starting backup at 24-SEP-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=+CRS_TMP/_MGMTDB/DATAFILE/sysmgmtdata.260.826899881
output file name=+MGMTDB/_MGMTDB/DATAFILE/sysmgmtdata.256.827005237 tag=TAG20130924T194034 RECID=1 STAMP=827005278
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:45
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=+CRS_TMP/_MGMTDB/DATAFILE/system.259.826899845
output file name=+MGMTDB/_MGMTDB/DATAFILE/system.257.827005281 tag=TAG20130924T194034 RECID=2 STAMP=827005291
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:16
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+CRS_TMP/_MGMTDB/DATAFILE/sysaux.258.826899821
output file name=+MGMTDB/_MGMTDB/DATAFILE/sysaux.258.827005295 tag=TAG20130924T194034 RECID=3 STAMP=827005303
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=+CRS_TMP/_MGMTDB/DATAFILE/sysgridhomedata.261.826899917
output file name=+MGMTDB/_MGMTDB/DATAFILE/sysgridhomedata.259.827005311 tag=TAG20130924T194034 RECID=4 STAMP=827005312
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=+CRS_TMP/_MGMTDB/DATAFILE/undotbs1.257.826899817
output file name=+MGMTDB/_MGMTDB/DATAFILE/undotbs1.260.827005313 tag=TAG20130924T194034 RECID=5 STAMP=827005316
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting datafile copy
copying current control file
output file name=+MGMTDB/_MGMTDB/CONTROLFILE/backup.261.827005317 tag=TAG20130924T194034 RECID=6 STAMP=827005317
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 24-SEP-13
channel ORA_DISK_1: finished piece 1 at 24-SEP-13
piece handle=+MGMTDB/_MGMTDB/BACKUPSET/2013_09_24/nnsnf0_tag20130924t194034_0.262.827005319 tag=TAG20130924T194034 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 24-SEP-13

RMAN> SWITCH DATABASE TO COPY;

datafile 1 switched to datafile copy "+MGMTDB/_MGMTDB/DATAFILE/system.257.827005281"
datafile 2 switched to datafile copy "+MGMTDB/_MGMTDB/DATAFILE/sysaux.258.827005295"
datafile 3 switched to datafile copy "+MGMTDB/_MGMTDB/DATAFILE/undotbs1.260.827005313"
datafile 4 switched to datafile copy "+MGMTDB/_MGMTDB/DATAFILE/sysgridhomedata.259.827005311"
datafile 5 switched to datafile copy "+MGMTDB/_MGMTDB/DATAFILE/sysmgmtdata.256.827005237"

Datafiles was migrated successful.

Checking Datafiles


RMAN> report schema;

Report of database schema for database with db_unique_name _MGMTDB

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    600      SYSTEM               ***     +MGMTDB/_MGMTDB/DATAFILE/system.257.827005281
2    400      SYSAUX               ***     +MGMTDB/_MGMTDB/DATAFILE/sysaux.258.827005295
3    50       UNDOTBS1             ***     +MGMTDB/_MGMTDB/DATAFILE/undotbs1.260.827005313
4    100      SYSGRIDHOMEDATA      ***     +MGMTDB/_MGMTDB/DATAFILE/sysgridhomedata.259.827005311
5    2048     SYSMGMTDATA          ***     +MGMTDB/_MGMTDB/DATAFILE/sysmgmtdata.256.827005237

List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    20       TEMP                 32767       +CRS_TMP/_MGMTDB/TEMPFILE/temp.266.826899969

Move Tempfile to Diskgroup MGMTDB.

RMAN> run {
SET NEWNAME FOR TEMPFILE 1 TO '+MGMTDB';
SWITCH TEMPFILE ALL;
 }

executing command: SET NEWNAME

renamed tempfile 1 to +MGMTDB in control file

At startup of database TEMP file will be recreated on new Diskgroup.

Starting Database and checking.

RMAN> startup

database is already started
database opened

RMAN> report schema;

using target database control file instead of recovery catalog
Report of database schema for database with db_unique_name _MGMTDB

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    600      SYSTEM               ***     +MGMTDB/_MGMTDB/DATAFILE/system.257.827005281
2    400      SYSAUX               ***     +MGMTDB/_MGMTDB/DATAFILE/sysaux.258.827005295
3    50       UNDOTBS1             ***     +MGMTDB/_MGMTDB/DATAFILE/undotbs1.260.827005313
4    100      SYSGRIDHOMEDATA      ***     +MGMTDB/_MGMTDB/DATAFILE/sysgridhomedata.259.827005311
5    2048     SYSMGMTDATA          ***     +MGMTDB/_MGMTDB/DATAFILE/sysmgmtdata.256.827005237

List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    20       TEMP                 32767       +MGMTDB/_MGMTDB/TEMPFILE/temp.261.827005913

RMAN> exit
Recovery Manager complete.

Delete the old datafiles copy on old Diskgroup CRS_TMP.

RMAN> delete copy ;

allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=124 device type=DISK
specification does not match any archived log in the repository
List of Datafile Copies
=======================

Key     File S Completion Time Ckp SCN    Ckp Time
------- ---- - --------------- ---------- ---------------
7       1    A 24-SEP-13       798766     24-SEP-13
        Name: +CRS_TMP/_MGMTDB/DATAFILE/system.259.826899845

8       2    A 24-SEP-13       798766     24-SEP-13
        Name: +CRS_TMP/_MGMTDB/DATAFILE/sysaux.258.826899821

9       3    A 24-SEP-13       798766     24-SEP-13
        Name: +CRS_TMP/_MGMTDB/DATAFILE/undotbs1.257.826899817

10      4    A 24-SEP-13       798766     24-SEP-13
        Name: +CRS_TMP/_MGMTDB/DATAFILE/sysgridhomedata.261.826899917

11      5    A 24-SEP-13       798766     24-SEP-13
        Name: +CRS_TMP/_MGMTDB/DATAFILE/sysmgmtdata.260.826899881

List of Control File Copies
===========================

Key     S Completion Time Ckp SCN    Ckp Time
------- - --------------- ---------- ---------------
6       A 24-SEP-13       798766     24-SEP-13
        Name: +MGMTDB/_MGMTDB/CONTROLFILE/backup.261.827005317
        Tag: TAG20130924T194034


Do you really want to delete the above objects (enter YES or NO)? yes
deleted datafile copy
datafile copy file name=+CRS_TMP/_MGMTDB/DATAFILE/system.259.826899845 RECID=7 STAMP=827005353
deleted datafile copy
datafile copy file name=+CRS_TMP/_MGMTDB/DATAFILE/sysaux.258.826899821 RECID=8 STAMP=827005353
deleted datafile copy
datafile copy file name=+CRS_TMP/_MGMTDB/DATAFILE/undotbs1.257.826899817 RECID=9 STAMP=827005353
deleted datafile copy
datafile copy file name=+CRS_TMP/_MGMTDB/DATAFILE/sysgridhomedata.261.826899917 RECID=10 STAMP=827005353
deleted datafile copy
datafile copy file name=+CRS_TMP/_MGMTDB/DATAFILE/sysmgmtdata.260.826899881 RECID=11 STAMP=827005353
deleted control file copy
control file copy file name=+MGMTDB/_MGMTDB/CONTROLFILE/backup.261.827005317 RECID=6 STAMP=827005317
Deleted 6 objects

Migrating Redo Logs to Diskgroup MGMTDB
P.S: Database must be OPENED.

[grid@node12c02 ~]$ sqlplus / as sysdba
SQL> select lf.group#,lf.member,lg.status from v$logfile lf, v$log lg where  lf.GROUP#=lg.GROUP#  order by 1;
    GROUP# MEMBER                                             STATUS
---------- -------------------------------------------------- ----------------
         1 +CRS_TMP/_MGMTDB/ONLINELOG/group_1.263.826899955   INACTIVE
         2 +CRS_TMP/_MGMTDB/ONLINELOG/group_2.264.826899955   CURRENT
         3 +CRS_TMP/_MGMTDB/ONLINELOG/group_3.265.826899957   INACTIVE

Adding New Log Members

SQL>  ALTER DATABASE ADD LOGFILE MEMBER '+MGMTDB' TO GROUP 1;
Database altered.

SQL>  ALTER DATABASE ADD LOGFILE MEMBER '+MGMTDB' TO GROUP 2;
Database altered.
 
SQL>  ALTER DATABASE ADD LOGFILE MEMBER '+MGMTDB' TO GROUP 3;
Database altered.

SQL> alter system switch logfile;

System altered.

SQL> /

System altered.

SQL> /

System altered.

SQL> /
System altered.

SQL> /
System altered.

SQL> alter system checkpoint;

Removing Log Member from CRS_TMP

SQL> select lf.group#,lf.member,lg.status from v$logfile lf, v$log lg where  lf.GROUP#=lg.GROUP#  order by 1;

    GROUP# MEMBER                                             STATUS
---------- -------------------------------------------------- ----------------
         1 +CRS_TMP/_MGMTDB/ONLINELOG/group_1.263.826899955   ACTIVE
         1 +MGMTDB/_MGMTDB/ONLINELOG/group_1.264.827006475    ACTIVE
         2 +MGMTDB/_MGMTDB/ONLINELOG/group_2.266.827006507    ACTIVE
         2 +CRS_TMP/_MGMTDB/ONLINELOG/group_2.264.826899955   ACTIVE
         3 +MGMTDB/_MGMTDB/ONLINELOG/group_3.265.827006479    CURRENT
         3 +CRS_TMP/_MGMTDB/ONLINELOG/group_3.265.826899957   CURRENT

6 rows selected.

SQL> ALTER DATABASE DROP LOGFILE MEMBER '+CRS_TMP/_MGMTDB/ONLINELOG/group_1.263.826899955';
Database altered.
 
SQL> ALTER DATABASE DROP LOGFILE MEMBER '+CRS_TMP/_MGMTDB/ONLINELOG/group_2.264.826899955';
Database altered.
 
SQL> ALTER DATABASE DROP LOGFILE MEMBER '+CRS_TMP/_MGMTDB/ONLINELOG/group_3.265.826899957'; 
Database altered.
		  
		 
		 
SQL> ALTER DATABASE DROP LOGFILE MEMBER '+CRS_TMP/_MGMTDB/ONLINELOG/group_3.265.826899957';
ALTER DATABASE DROP LOGFILE MEMBER '+CRS_TMP/_MGMTDB/ONLINELOG/group_3.265.826899957'
*
ERROR at line 1:
ORA-01609: log 3 is the current log for thread 1 - cannot drop members
ORA-00312: online log 3 thread 1:
'+CRS_TMP/_MGMTDB/ONLINELOG/group_3.265.826899957'
ORA-00312: online log 3 thread 1:
'+MGMTDB/_MGMTDB/ONLINELOG/group_3.265.827006479'

SQL> alter system switch logfile;

System altered.

SQL> alter system checkpoint;

System altered.

SQL> ALTER DATABASE DROP LOGFILE MEMBER '+CRS_TMP/_MGMTDB/ONLINELOG/group_3.265.826899957';

Database altered.

SQL> select lf.group#,lf.member,lg.status from v$logfile lf, v$log lg where  lf.GROUP#=lg.GROUP#  order by 1;

    GROUP# MEMBER                                             STATUS
---------- -------------------------------------------------- ----------------
         1 +MGMTDB/_MGMTDB/ONLINELOG/group_1.264.827006475    CURRENT
         2 +MGMTDB/_MGMTDB/ONLINELOG/group_2.266.827006507    INACTIVE
         3 +MGMTDB/_MGMTDB/ONLINELOG/group_3.265.827006479    INACTIVE

Redo Logs was moved successful.

Check Configuration of resource ora.mgmtdb.
Maybe you need modify this resource by removing old DISKGROUP.

As grid user:

[grid@node12c02 ~]$ crsctl status res ora.mgmtdb -p |grep CRS_TMP
START_DEPENDENCIES=hard(ora.MGMTLSNR,ora.CRS_TMP.dg,ora.MGMTDB.dg) weak(uniform:ora.ons) pullup(ora.MGMTLSNR,ora.CRS_TMP.dg,ora.MGMTDB.dg)
STOP_DEPENDENCIES=hard(intermediate:ora.MGMTLSNR,shutdown:ora.CRS_TMP.dg,intermediate:ora.asm,shutdown:ora.MGMTDB.dg)

If command above return output you need update OCR by removing the old Diskgroup.
If you removed Diskgroup without update resource ora.mgmtdb, srvctl will not be able to start resource ora.mgmtdb and raise error:

[grid@node12c02 ~]$ srvctl config mgmtdb
Database unique name: _mgmtdb
Database name:
Oracle home: /u01/app/12.1.0/grid
Oracle user: grid
Spfile: +MGMTDB/_MGMTDB/PARAMETERFILE/spfile.256.828288969
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Database instance: -MGMTDB
PRCD-1012 : Failed to retrieve disk group list for database _mgmtdb.
PRCR-1035 : Failed to look up CRS resource ora.CRS_TMP.dg for _mgmtdb
PRCA-1000 : ASM Disk Group CRS_TMP does not exist
PRCR-1001 : Resource ora.CRS_TMP.dg does not exist

To fix it you need remove this resource.
If resource ora.CRS_TMP.dg does not exists you must use crsctl with force option ("-f" flag) to modify ora.mgmtdb resource.

[grid@node12c02 ~]$ crsctl modify res ora.mgmtdb -attr "START_DEPENDENCIES='hard(ora.MGMTLSNR,ora.MGMTDB.dg) weak(uniform:ora.ons) pullup(ora.MGMTLSNR,ora.MGMTDB.dg)'"
CRS-2510: Resource 'ora.CRS_TMP.dg' used in dependency 'hard' does not exist or is not registered
CRS-2514: Dependency attribute specification 'hard' is invalid in resource 'ora.mgmtdb'
CRS-4000: Command Modify failed, or completed with errors.

Updating resource ora.mgmtdb by removing ora.CRS_TMP.dg resource.


[grid@node12c02 ~]$ crsctl modify res ora.mgmtdb -attr "START_DEPENDENCIES='hard(ora.MGMTLSNR,ora.MGMTDB.dg) weak(uniform:ora.ons) pullup(ora.MGMTLSNR,ora.MGMTDB.dg)'" -f

[grid@node12c02 ~]$ crsctl modify res ora.mgmtdb -attr "STOP_DEPENDENCIES='hard(intermediate:ora.MGMTLSNR,intermediate:ora.asm,shutdown:ora.MGMTDB.dg)'" -f

[grid@node12c02 ~]$ crsctl status res ora.mgmtdb -p |grep CRS_TMP

[grid@node12c02 ~]$ srvctl config mgmtdb
Database unique name: _mgmtdb
Database name:
Oracle home: /u01/app/12.1.0/grid
Oracle user: grid
Spfile: +MGMTDB/_MGMTDB/PARAMETERFILE/spfile.256.828288969
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Database instance: -MGMTDB
Type: Management

[grid@node12c02 ~]$ srvctl start mgmtdb

[grid@node12c02 ~]$ srvctl status mgmtdb
Database is enabled
Instance -MGMTDB is running on node node12c02 

Adding OCR Mirror on +MGMTDB.

[root@node12c02 ~]# /u01/app/12.1.0/grid/bin/ocrconfig -add +MGMTDB
[root@node12c02 ~]# /u01/app/12.1.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1568
         Available space (kbytes) :     408000
         ID                       : 1037914462
         Device/File Name         :  +OCR_VOTE
                                    Device/File integrity check succeeded
         Device/File Name         :    +MGMTDB
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

I recommend stop and start your cluster by issuing (as root) "crsctl stop cluster -all and crsctl start cluster -all"

If all resources restart successfull, then we can stop diskgroup "CRS_TMP".

[grid@node12c02 ~]$ srvctl stop diskgroup -g CRS_TMP

[grid@node12c02 ~]$ srvctl status diskgroup -g CRS_TMP
Disk Group CRS_TMP is not running

Check if all resources is online. If yes, you can remove this diskgroup.

[grid@node12c02 ~]$ srvctl start diskgroup -g CRS_TMP -n node12c02 

[grid@node12c02 ~]$ export ORACLE_SID=+ASM1
[grid@node12c02 ~]$ sqlplus / as sysasm

SQL> DROP DISKGROUP CRS_TMP INCLUDING CONTENTS;

Diskgroup dropped.

Then restart whole clusterware again and check if all resources started without issue.


Oracle Database 12c Release 1 (12.1) New Features


Oracle Database 12c Release 1 (12.1) New Features
This is the main page of documentation on Oracle 12c, is highly recommended for all 11g Database Administrators.

  • Application Development
  • Business Intelligence and Data Warehousing
  • Compression and Archiving
  • Database Overall
  • High Availability
  • Manageability
  • Performance
  • Oracle RAC and Grid Infrastructure
  • Security
  • Spatial and Graph
  • Unstructured Data
  • Upgrades
  • Windows

    .
    See all New Features Here


  • How to fast recover data lost due to user errors (without RMAN)

    To a DBA time is money. In cases of downtime …. more time stoped, less money and less time stoped, more money.

    Many times we make mistakes for not knowing all the resources available in certain situations that require quick action.

    When someone calls you and says that a someone mistakenly deleted data, usually the first thought that comes is that we need to perform the restore of the data.

    Here comes an important issue. How to restore data in a short period to cause less impact and less downtime.

    Below is the request of a colleague who was in this situation, read:
    Version:10g R2
    One of our DBA colleagues accidently updated all the records of a table to a particular values without a WHERE clause.I need to revert it back to its previous state (as of 11 am today morning)
    Since this is 3 TB schema, we don’t take logical backup for this schema. Flashback feauture is not enabled either.
    I have LEVEL0 backup from Monday morning and LEVEL1 backup from Tuesday (today) morning. How can I restore the table to the previous state?

    We have many options, but what option you decide to use will make all difference.

    The first feature that I will try to use is the FLASHBACK QUERY.  Although this feature exists since version 9i unfortunately for many people is still a novelty.

    Let’s test:

    $ sqlplus user_test@db11g
    
    SQL*Plus: Release 11.2.0.2.0 Production on Fri Feb 3 14:32:56 2012
    Copyright (c) 1982, 2009, Oracle.  All rights reserved.
    Enter password:
    
    Connected to:
    Oracle Database 11g Release 11.2.0.1.0 - Production
    With the Automatic Storage Management option
    
    SQL> select flashback_on from v$database;
    
    FLASHBACK_ON
    ------------------
    NO
    
    ### CREATING TABLE############
    SQL>
    CREATE TABLE USER_ERROR AS SELECT * FROM DBA_OBJECTS;
    Table created.
    ##############################
    
    ### Populating Data ##########
    INSERT INTO USER_ERROR SELECT * FROM USER_ERROR;
    COMMIT;
    76416 rows inserted
    commited
    
    INSERT INTO USER_ERROR SELECT * FROM USER_ERROR;
    COMMIT;
    152832 rows inserted
    commited
    
    INSERT INTO USER_ERROR SELECT * FROM USER_ERROR;
    COMMIT;
    305664 rows inserted
    commited
    
    INSERT INTO USER_ERROR SELECT * FROM USER_ERROR;
    COMMIT;
    611328 rows inserted
    commited
    ##############################
    
    ### Checking Data
    SELECT COUNT(1) FROM USER_ERROR;
    
    COUNT(1)
    ----------------------
    1222656
    
    SQL> select count(object_type) qtd,object_type
    from user_error
    group by object_type
    order by 1;
    
           QTD OBJECT_TYPE
    ---------- -------------------
            16 LOB PARTITION
            16 EDITION
            16 MATERIALIZED VIEW
            16 RULE
            32 DESTINATION
            48 JAVA SOURCE
            48 SCHEDULE
            64 SCHEDULER GROUP
           112 CONTEXT
           144 INDEXTYPE
           144 WINDOW
           144 UNDEFINED
           160 RESOURCE PLAN
           160 CLUSTER
           208 EVALUATION CONTEXT
           208 JOB CLASS
           256 DIRECTORY
           304 RULE SET
           304 PROGRAM
           400 CONSUMER GROUP
           448 JOB
           576 QUEUE
           816 XML SCHEMA
           880 OPERATOR
          2928 LIBRARY
          3552 PROCEDURE
          3824 TYPE BODY
          3984 SEQUENCE
          4448 TABLE PARTITION
          4864 INDEX PARTITION
          4880 JAVA DATA
          5392 FUNCTION
         13344 JAVA RESOURCE
         17168 TRIGGER
         17296 LOB
         23680 PACKAGE BODY
         24752 PACKAGE
         44624 TYPE
         58752 TABLE
         84384 INDEX
         86384 VIEW
        366768 JAVA CLASS
        446112 SYNONYM
    
    43 rows selected.
    
    SQL> SELECT CURRENT_TIMESTAMP FROM DUAL;
    
    CURRENT_TIMESTAMP
    ---------------------------------------------------------------------------
    03-FEB-12 02.38.08.300885 PM -02:00
    
    ### UPDATE WITHOUT WHERE CLAUSE
    
    SQL> UPDATE USER_ERROR SET OBJECT_TYPE='UNKNOWN';
    
    1222656 rows updated.
    
    SQL> COMMIT;
    
    Commit complete.
    
    SQL> select count(object_type) qtd,object_type
    from user_error
    group by object_type
    order by 1;
    
           QTD OBJECT_TYPE
    ---------- -------------------
       1222656 UNKNOWN
    
    ###### RECOVERING DATA QUICKLY #####
    
    SQL>  CREATE TABLE USER_ERROR_RECOVERED
          AS
          SELECT * FROM USER_ERROR
          AS OF TIMESTAMP TO_TIMESTAMP('03-02-2012 14:38:08','DD-MM-YYYY HH24:MI:SS');
    
    Table created.
    
    SQL> SELECT COUNT(1) FROM USER_ERROR_RECOVERED;
    
      COUNT(1)
    ----------
       1222656
    
    SQL> select count(object_type) qtd,object_type
    from user_error_recovered
    group by object_type
    order by 1;
           QTD OBJECT_TYPE
    ---------- -------------------
            16 RULE
            16 LOB PARTITION
            16 MATERIALIZED VIEW
            16 EDITION
            32 DESTINATION
            48 JAVA SOURCE
            48 SCHEDULE
            64 SCHEDULER GROUP
           112 CONTEXT
           144 UNDEFINED
           144 WINDOW
           144 INDEXTYPE
           160 CLUSTER
           160 RESOURCE PLAN
           208 JOB CLASS
           208 EVALUATION CONTEXT
           256 DIRECTORY
           304 PROGRAM
           304 RULE SET
           400 CONSUMER GROUP
           448 JOB
           576 QUEUE
           816 XML SCHEMA
           880 OPERATOR
          2928 LIBRARY
          3552 PROCEDURE
          3824 TYPE BODY
          3984 SEQUENCE
          4448 TABLE PARTITION
          4864 INDEX PARTITION
          4880 JAVA DATA
          5392 FUNCTION
         13344 JAVA RESOURCE
         17168 TRIGGER
         17296 LOB
         23680 PACKAGE BODY
         24752 PACKAGE
         44624 TYPE
         58752 TABLE
         84384 INDEX
         86384 VIEW
        366768 JAVA CLASS
        446112 SYNONYM
    
    43 rows selected.
    
    ### DROPPING TABLE #######
    
    SQL> DROP TABLE USER_ERROR;
    
    Table dropped.
    
    SQL> SELECT COUNT(1) FROM USER_ERROR;
    SELECT COUNT(1) FROM USER_ERROR
                         *
    ERROR at line 1:
    ORA-00942: table or view does not exist
    
    SQL> FLASHBACK TABLE USER_ERROR TO BEFORE DROP;
    
    Flashback complete.
    
    SQL>  SELECT COUNT(1) FROM USER_ERROR;
    
      COUNT(1)
    ----------
       1222656
    
    SQL>
    

    In the example above I was able to restore data in a short time, saving work and time.

    For this procedure succeeds the data must still be in the UNDO tablespace, then when more quickly you identify the error and try to fix it, Will increase the probability of success.

    Oracle used the term “flashback” to cover very different things, I don’t like this because it confuse which feature we can use.

    To use Flasback Table or Flasback Query you don’t need the “flashback feature” enabled, because flashback query is based on undo information, unlike flashback database which needs flashback logs + redo logs.

    What Oracle Editions I can use feature flasback query?
    Flashback Query – Is enabled to use in all Oracle Editions no additional cost.
    http://www.oracle.com/us/products/database/product-editions-066501.html

    What Oracle Editions I can use feature flasback table?
    Flashback Table – Is enabled to use only in Oracle Enterprise Edition. (Works in SE edition)
    http://www.oracle.com/us/products/database/product-editions-066501.html

    Enjoy…

    Google


    News: Now is Supported BACKUPSET/ARCHIVELOG/DUMPSET files on ACFS 11.2.0.3

    Starting with Oracle Automatic Storage Management 11g Release 2 (11.2.0.3), Oracle ACFS supports RMAN backups (BACKUPSET file type), archive logs (ARCHIVELOG file type), and Data Pump dumpsets (DUMPSET file type). Note that Oracle ACFS snapshots are not supported with these files.

     

    ASM best practices we recommend to have two ASM disk groups:

    DATAOH Diskgroup contains:
    • Database files (Database files must be stored directly in ASM and not on ACFS)
    • Oracle Clusterware files (OCR/Vote files)
    • ASM Spfile
    • ADVM volumes/ACFS filesystems for database related data or general-purpose use

     

    FRA Diskgroup

    This disk group will be used to store database recovery related files; such as archived log files, RMAN backups, and Flashback logs

    • Datapump dumpsets , RMAN backup files, etc. This was supported in ACFS 11.2.0.3 and above; however, ACFS does not currently support snapshots of filesystems housing  these files.
    • Additionally, database ORACLE_HOME backups (possibly zipped backups), can stored in a filesystem carved from the FRA disk group
    ACFS Technical Overview and Deployment Guide [ID 948187.1]

    http://docs.oracle.com/cd/E11882_01/server.112/e18951/asmfilesystem.htm#CACJFGCD

     

    Enjoy…

     

     


    How configure Multiples Public Network in a Grid Infrastructure 11g R2 (11.2) environment

    Introduction

    Using earlier versions of RAC Oracle 11.2 is not possible to create more than one public subnet to serve different database clients in different sites/network, to get the routing of connections we should have help from a software/hardware from third parties.

    Now it’s possible configure multiples network segment (i.e different subnet) with GI/RAC 11.2.
    Oracle GI/RAC 11.2 works like a router managing connections of multiple network, routing connections to the correct network/endpoint.

    During the Grid Infrastructure installation, it’s not possible configure more than one public/vip network. So, the other networks should be configured manually after installation.

    Purpose

    The purpose of this post is to explain how to create  multipe network/listener/services on multiple public network in an 11.2 Grid Infrastructure environment.

    No downtime is required to perform this task. All tasks can be done with Clusterware and RAC online.

    Concept before Starting

    Please Note – Limitations of Grid Infrastructure 11g R2:

    SCAN feature will work only in one public network, because we can configure only one SCAN for entire Clusterware Stack and SCAN must be associated whith only one public network. So, only one  public network  we can use all feature  of GI 11.2 the others public networks will be required use the VIP in our Oracle Clients  like in version 11g R1 and 10g.

    I believe that Oracle will solve this issue in later versions.

    Before start configuration you must learn how things work to know to properly configure our RAC.  You need to know how they work to understand how to configure your Oracle RAC.


    Basic Concept:

    The listener is a server-side process that listens for incoming client connection requests and manages traffic to the database. When a database instance starts, and at various times during its life, the instance contacts a listener and establishes a communication pathway to this instance.

    Service registration enables the listener to determine whether a database service and its service handlers are available. A service handler is a dedicated server process or dispatcher that acts as a connection point to a database. During registration, the PMON process provides the listener with the instance name, database service names, and the type and addresses of service handlers. This information enables the listener to start a service handler when a client request arrives.

    A service name is a logical representation of a service used for client connections.

    When a client connects to a listener, it requests a connection to a service.Thus, the listener acts as a mediator between the client and instances and routes the connection request to the right place.

    To ensure service registration works properly,  the initialization parameter file should contain the following parameters:

    • LOCAL_LISTENER for the local listener
    • REMOTE_LISTENER for the remote listener

    LOCAL_LISTENER: To have PMON register with a local listener in RAC env we must configure the LOCAL_LISTENER parameter in the initialization parameter file to locate the local listener. Multiple addresses are supported, but connect-time failover and client load balancing features are not supported.

    REMOTE_LISTENER: A remote listener is a listener residing on one computer that redirects connections to a database instance on another computer. In a dedicated server RAC environment, we must enable the PMON background process to register with a remote listener. You do this by configuring the REMOTE_LISTENER parameter.

    Each network must have a dedicated LISTENER.  It would be necessary to configure multiple addresses (Listener) in the parameter  LOCAL_LISTENER from all instances  to serve NET1 and NET2. Multiple addresses are supported, but connect-time failover and client load balancing features are not supported. (first problem) 😦

    If we use  only parameter LOCAL_LISTENER and REMOTE_LISTENER the connections can be redirected by REMOTE_LISTENER to LOCAL_LISTENER  from others network. (second problem) 😦 😦

    To ensure that connections to the remote listener are only redirected to the local listener on the same network Oracle create new Parameter LISTENER_NETWORKS :-), this parameter is avaliable only in version 11g R2.

    • LISTENER_NETWORKS specifies one or more sets of local & remote listeners for cross-registration. All listeners within the same network_name will cross-register.

    Using parameter LISTENER_NETWORKS we can configure multipes LOCAL & REMOTE Listener in your own network.  It ensure wich connections incoming by network net1 will be redirected over RAC Database only on network net1.

    This tasks will be done in 2 steps:

    1. Create New Network and Listener in Oracle Clusterware

    2. Configure RAC to support multipe Network and Create Service

    Creating  New Network and Listener in Oracle Clusterware

    The Clusterware must be configured and online in all nodes.

    Overview of Environment

    1. First Public network is called North.
    2. Second Public network is called South.
    3. Third Network is called East.
    Infrastructure:
    
    Servers Hostname
    1° northora01
    2° northora02
    3° northora03
    
    North - eth0 - 192.168.217.0
    # Public Hostnames/IP North
    northora01 - 192.168.217.60
    northora02 - 192.168.217.45
    northora03 - 192.168.217.68
    
    # Virtual Hostnames/IP North
    northora01-vip - 192.168.217.52
    northora02-vip - 192.168.217.53
    northora02-vip - 192.168.217.69
    
    South - eth2 - 140.120.120.0
    # Public Hostnames/IP South
    southora01 - 140.120.120.200
    southora02 - 140.120.120.201
    southora03 - 140.120.120.202
    # Virtual Hostnames/IP South
    southora01-vip - 140.120.120.100
    southora02-vip - 140.120.120.101
    southora03-vip - 140.120.120.102
    
    East - eth3 - 154.120.120.0
    # Public Hostnames/IP East
    eastora01 - 154.120.120.200
    eastora02 - 154.120.120.201
    eastora03 - 154.120.120.202
    # Virtual Hostnames/IP East
    eastora01-vip - 154.120.120.100
    eastora02-vip - 154.120.120.101
    eastora03-vip - 154.120.120.102
    

    Before start the New Interfaces (eth2 and eth3) must be configured with IP,Mask and Gateway.

    After setting up the new interfaces you must set VIP hostnames from new networks in hosts files (e.g /etc/hosts) from all nodes and add VIP-hostnames in your DNS.

    # /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1               localhost.localdomain localhost
    ::1             localhost6.localdomain6 localhost6
    # Public North
    192.168.217.60          northora01.north.com		northora01
    192.168.217.45          northora02.north.com		northora02
    192.168.217.68          northora03.north.com		northora03
    
    # VIP North
    192.168.217.52          northora01-vip.north.com		northora01-vip
    192.168.217.53          northora02-vip.north.com		northora02-vip
    192.168.217.69          northora03-vip.north.com		northora03-vip
    
    # Public South
    140.120.120.200         southora01.south.com            southora01
    140.120.120.201         southora02.south.com            southora02
    140.120.120.202         southora03.south.com            southora03
    
    # VIP South
    140.120.120.100         southora01-vip.south.com	southora01-vip
    140.120.120.101         southora02-vip.south.com	southora02-vip
    140.120.120.102        	southora03-vip.south.com	southora03-vip
    
    # Public East
    154.120.120.200		eastora01.east.com		eastora01
    154.120.120.201		eastora02.east.com		eastora02
    154.120.120.202		eastora03.east.com		eastora03
    
    # Vip East
    154.120.120.100		eastora01-vip.east.com		eastora01-vip
    154.120.120.101		eastora02-vip.east.com		eastora02-vip
    154.120.120.102		eastora03-vip.east.com		eastora03-vip
    


    Ensure Public and VIP address is defined for the 2nd and 3nd public network

    Creating the CRS resource (network/vip) using srvctl

    A new network and new vip resources for the new network are created by using the ‘-k’ switch.
    The-k option indicates to which network it is. The public network has automatically installed the first number.

    Using Oracle Grid Infrastucture 11.2.0.1 you can’t create network resource explicitly, when we create a new vip resource the network resource will be created implicitly and a dependency between resources (vip and network) will be configured.

    Retrieving information about the environment

    # Actual Network
    # Network Info
    crsctl status resource -f |grep NAME=ora.net -A 3
    NAME=ora.net1.network
    TYPE=ora.network.type
    STATE=ONLINE
    TARGET=ONLINE
    
    # VIP Info
    crsctl status resource -f |grep NAME=ora.northora -A 3 |grep .vip -A 3
    NAME=ora.northora01.vip
    TYPE=ora.cluster_vip_net1.type
    STATE=ONLINE
    TARGET=ONLINE
    --
    NAME=ora.northora02.vip
    TYPE=ora.cluster_vip_net1.type
    STATE=ONLINE
    TARGET=ONLINE
    --
    NAME=ora.northora03.vip
    TYPE=ora.cluster_vip_net1.type
    STATE=ONLINE
    TARGET=ONLINE
    
    # Getting Nodes Information
    olsnodes -n -s
    northora01	1       Active
    northora02  	2       Active
    northora03   	3       Active
    

    Adding Network/VIP Resources using Clusterware 11.2.0.1

    as root user:
    
    # srvctl add vip -n  -k  -A //[if1[|if2...]] [-v]
    
    srvctl add vip -n northora01 -A southora01-vip/255.255.255.0/eth2 -k 2
    srvctl add vip -n northora02 -A southora02-vip/255.255.255.0/eth2 -k 2
    srvctl add vip -n northora03 -A southora03-vip/255.255.255.0/eth2 -k 2
    
    srvctl start vip -i southora01-vip
    srvctl start vip -i southora02-vip
    srvctl start vip -i southora03-vip
    
    # Checking Status Network 2
    crsctl status resource -f |grep NAME=ora.net2 -A 3
    NAME=ora.net2.network
    TYPE=ora.network.type
    STATE=ONLINE
    TARGET=ONLINE
    
    crsctl status resource -f |grep NAME=ora.south -A 3 |grep .vip -A 3
    NAME=ora.southora01-vip.vip
    TYPE=ora.cluster_vip_net2.type
    STATE=ONLINE
    TARGET=ONLINE
    --
    NAME=ora.ora.southora02-vip.vip.vip
    TYPE=ora.cluster_vip_net2.type
    STATE=ONLINE
    TARGET=ONLINE
    --
    NAME=ora.ora.southora03-vip.vip.vip
    TYPE=ora.cluster_vip_net2.type
    STATE=ONLINE
    TARGET=ONLINE
    

    From 11.2.0.2+, network resource can be created explicitly:

    as root user:
    # srvctl add network [-k ] -S //[if1[|if2...]] [-w ] [-v]
    
    srvctl add network -k 3 -S 154.120.120.0/255.255.255.0/eth3
    
    crsctl status resource -f |grep NAME=ora.net3 -A 3
    NAME=ora.net3.network
    TYPE=ora.network.type
    STATE=OFFLINE
    TARGET=OFFLINE
    
    Then add vip resource for the 3nd network:
    srvctl add vip -n northora01 -A eastora01-vip/255.255.255.0/eth3 -k 3
    srvctl add vip -n northora02 -A eastora02-vip/255.255.255.0/eth3 -k 3
    srvctl add vip -n northora03 -A eastora03-vip/255.255.255.0/eth3 -k 3
    
    And starting vip resource
    srvctl start vip -i eastora01-vip
    srvctl start vip -i eastora02-vip
    srvctl start vip -i eastora03-vip
    
     crsctl status resource -f |grep NAME=ora.east -A 3 |grep .vip -A 3
    NAME=ora.eastora01-vip.vip
    TYPE=ora.cluster_vip_net3.type
    STATE=ONLINE
    TARGET=ONLINE
    --
    NAME=ora.eastora02-vip.vip
    TYPE=ora.cluster_vip_net3.type
    STATE=ONLINE
    TARGET=ONLINE
    --
    NAME=ora.eastora03-vip.vip
    TYPE=ora.cluster_vip_net3.type
    STATE=ONLINE
    TARGET=ONLINE
    

    Creating New Listener  for each network.

    You can create new listener using srvctl, but I recommend create Listener using NETCA.
    As the grid user invoke “netca” from the 11.2 GRID_HOME, the select “Cluster configuration” -> “Listener configuration” -> “Add”, enter Listener name as required.

    Repeat the procedure above to create the Listener on the network South.

    Validating Clusterware Configuration

    Checking Listeners created

    srvctl config listener -a
    Name: LISTENER
    Network: 1, Owner: oracle
    Home:
      /u01/app/11.2.0/grid on node(s) northora01,northora02,northora03
    End points: TCP:1521
    Name: LISTENER_SOUTH
    Network: 2, Owner: oracle
    Home:
      /u01/app/11.2.0/grid on node(s) northora01,northora02,northora03
    End points: TCP:1522
    Name: LISTENER_EAST
    Network: 3, Owner: oracle
    Home:
      /u01/app/11.2.0/grid on node(s) northora01,northora02,northora03
    End points: TCP:1523
    
    srvctl status listener
    Listener LISTENER is enabled
    Listener LISTENER is running on node(s): northora01,northora02,northora03
    Listener LISTENER_EAST is enabled
    Listener LISTENER_EAST is running on node(s): northora01,northora02,northora03
    Listener LISTENER_SOUTH is enabled
    Listener LISTENER_SOUTH is running on node(s): northora01,northora02,northora03
    
    srvctl status scan_listener
    SCAN Listener LISTENER_SCAN1 is enabled
    SCAN listener LISTENER_SCAN1 is running on node northora01
    SCAN Listener LISTENER_SCAN2 is enabled
    SCAN listener LISTENER_SCAN2 is running on node northora02
    SCAN Listener LISTENER_SCAN3 is enabled
    SCAN listener LISTENER_SCAN3 is running on node northora03
    
    srvctl config scan_listener
    SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
    SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
    SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521
    
    

    The Configuration of  network and listener on Cluster-Wide is finished.

    Configuring RAC Database “db11g”  to support multipe Network and Creating Service

    Now we need configure Oracle RAC Database (11.2) to provide service on all networks (network 1,2 and 3).

     Prepare the database instance for the new listener

    To ensure that connections to the remote listener are only redirected to the local listener on the same network, LISTENER_NETWORKS parameter needs to be set in the pfile or spfile for the database instance.

    Use tnsnames.ora from Oracle Home (RAC)  to resolve listener name alias for LISTENER_NETWORKS.  All alias must be in tnsnames.ora from all nodes of RAC Database.

    # tnsnames.ora Network Configuration File: /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames.ora
    # Generated by Oracle configuration tools.
    LISTENER_NORTH1=
     (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = northora01-vip.oracle.com)(PORT = 1521))
      )
    
    LISTENER_NORTH2=
     (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = northora02-vip.oracle.com)(PORT = 1521))
      )
    
    LISTENER_NORTH3=
     (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = northora03-vip.oracle.com)(PORT = 1521))
      )
    
    LISTENER_SOUTH1 =
      (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = southora01-vip.oracle.com)(PORT = 1522))
      )
    
    LISTENER_SOUTH2 =
      (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = southora02-vip.oracle.com)(PORT = 1522))
      )
    
    LISTENER_SOUTH3 =
      (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = southora03-vip.oracle.com)(PORT = 1522))
      )
    
    REMOTE_SOUTH =
       (DESCRIPTION_LIST =
         (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = southora01-vip.oracle.com)(PORT = 1522)))
         (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = southora02-vip.oracle.com)(PORT = 1522)))
         (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = southora03-vip.oracle.com)(PORT = 1522)))
       )
    
    LISTENER_EAST1 =
      (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = eastora01-vip.east.com)(PORT = 1523))
      )
    
    LISTENER_EAST2 =
      (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = eastora02-vip.east.com)(PORT = 1523))
      )
    
    LISTENER_EAST3 =
      (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = eastora03-vip.east.com)(PORT = 1523))
      )
    
    REMOTE_EAST =
       (DESCRIPTION_LIST =
         (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora01-vip.east.com)(PORT = 1523)))
         (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora02-vip.east.com)(PORT = 1523)))
         (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora03-vip.east.com)(PORT = 1523)))
       )
    
    

    Don’t forget configure sqlnet.ora.

    # sqlnet.ora Network Configuration File: /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.ora
    # Generated by Oracle configuration tools.
    
    NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
    
    ADR_BASE = /u01/app/oracle
    

    Testing connectivity of all entries in TNSNAMES.ora using utility tnsping

    This step is very important, all of listener alias name must be validated in all nodes.

    e.g

    tnsping LISTENER_SOUTH1
    
    TNS Ping Utility for Linux: Version 11.2.0.2.0 - Production on 18-OCT-2011 17:31:12
    
    Copyright (c) 1997, 2010, Oracle.  All rights reserved.
    
    Used parameter files:
    /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.ora
    
    Used TNSNAMES adapter to resolve the alias
    Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = southora1-vip.oracle.com)(PORT = 1522)))
    OK (0 msec)
    
    # Testing REMOTE
    
    tnsping REMOTE_EAST
    
    TNS Ping Utility for Linux: Version 11.2.0.2.0 - Production on 18-OCT-2011 17:31:55
    
    Copyright (c) 1997, 2010, Oracle.  All rights reserved.
    
    Used parameter files:
    /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.ora
    
    Used TNSNAMES adapter to resolve the alias
    Attempting to contact (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora01-vip.east.com)(PORT = 1523))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora02-vip.east.com)(PORT = 1523))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora03-vip.east.com)(PORT = 1523))))
    OK (0 msec)
    

    Configuring Oracle RAC to support multiples network

    Please, don’t change  current parameter of LOCAL_LISTENER and REMOTE_LISTENER of your spfile/pfile.

    Setting  LISTENER_NETWORKS is enough to RAC to work properly.

    Recommendation : Set parameter using scope=both, it’s will validade if the all Listener alias is working. If you use scope=spfile, it will not validaded and if not was configured properly the error will be raised at moment of startup of database instance.

    ####### Instance 1 #############
    ALTER SYSTEM
     SET LISTENER_NETWORKS = '((NAME=NORTH_NET)(LOCAL_LISTENER=LISTENER_NORTH1)(REMOTE_LISTENER=global-scan.north.com:1521))',
                             '((NAME=SOUTH_NET)(LOCAL_LISTENER=LISTENER_SOUTH1)(REMOTE_LISTENER=REMOTE_SOUTH))',
                             '((NAME=EAST_NET)(LOCAL_LISTENER=LISTENER_EAST1)(REMOTE_LISTENER=REMOTE_EAST))'
     SCOPE=BOTH SID='db11g1';
    ####### Instance 2 #############
    System altered.
    
    ALTER SYSTEM
     SET LISTENER_NETWORKS='((NAME=NORTH_NET)(LOCAL_LISTENER=LISTENER_NORTH2)(REMOTE_LISTENER=global-scan.north.com:1521))',
                           '((NAME=SOUTH_NET)(LOCAL_LISTENER=LISTENER_SOUTH2)(REMOTE_LISTENER=REMOTE_SOUTH))',
                           '((NAME=EAST_NET)(LOCAL_LISTENER=LISTENER_EAST2)(REMOTE_LISTENER=REMOTE_EAST))'
    SCOPE=BOTH SID='db11g2';
    
    System altered.
    
    ####### Instance 3 #############
    ALTER SYSTEM
     SET LISTENER_NETWORKS='((NAME=NORTH_NET)(LOCAL_LISTENER=LISTENER_NORTH3)(REMOTE_LISTENER=global-scan.north.com:1521))',
                           '((NAME=SOUTH_NET)(LOCAL_LISTENER=LISTENER_SOUTH3)(REMOTE_LISTENER=REMOTE_SOUTH))',
                           '((NAME=EAST_NET)(LOCAL_LISTENER=LISTENER_EAST3)(REMOTE_LISTENER=REMOTE_EAST))'
    SCOPE=BOTH SID='db11g3';
    
    System altered.
    

    Checking if default database service db11g was registered proprely

    #################### Network 1 #####################
    
    lsnrctl status LISTENER
    ..
    .
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.52)(PORT=1521)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.60)(PORT=1521)))
    Services Summary...
    ..
    .
    Service "db11g" has 1 instance(s).
      Instance "db11g2", status READY, has 1 handler(s) for this service...
    ..
    .
    The command completed successfully
    
    # In network 1 the remote_listener is registered in Listener SCAN
    lsnrctl status LISTENER_SCAN3
    ..
    .
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN3)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.62)(PORT=1521)))
    Services Summary...
    ..
    .
    Service "db11g" has 3 instance(s).
      Instance "db11g1", status READY, has 2 handler(s) for this service...
      Instance "db11g2", status READY, has 2 handler(s) for this service...
      Instance "db11g3", status READY, has 2 handler(s) for this service...
    ..
    .
    The command completed successfully
    
    #################### Network 2 #####################
    
    lsnrctl status LISTENER_SOUTH
    ..
    .
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SOUTH)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=140.120.120.101)(PORT=1522)))
    Services Summary...
    ..
    .
    Service "db11g" has 3 instance(s).
      Instance "db11g1", status READY, has 1 handler(s) for this service...
      Instance "db11g2", status READY, has 2 handler(s) for this service...
      Instance "db11g3", status READY, has 1 handler(s) for this service...
    The command completed successfully
    
    #################### Network 2 #####################
    
    lsnrctl status LISTENER_EAST
    ..
    .
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_EAST)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=154.120.120.101)(PORT=1523)))
    Services Summary...
    ..
    .
    Service "db11g" has 3 instance(s).
      Instance "db11g1", status READY, has 1 handler(s) for this service...
      Instance "db11g2", status READY, has 2 handler(s) for this service...
      Instance "db11g3", status READY, has 1 handler(s) for this service...
    The command completed successfully
    

    Creating Service “db11g_inbound”

    Using “srvctl add service” with option -k it’s possible create a dependency specifying the number of network. However, this has no effect on listeners which is registered with the service, the service will be registered in all listener configured in RAC (LOCAL,REMOTE and NETWORK Listener). It rather serves only to ensure that the service is started, when the corresponding subnet, or the corresponding ora.netX.network resource is present and started.
    Bad news, each service can be configure with  dependency  to only one network. We can not specify multiple networks numbers.

    I created a workaround to fix it. (It’s not supported by Oracle)

    Creating new Service DB11G_INBOUND

    In example below a dependency will be created with ora.net2.network

    srvctl add service -s db11g_inbound -m BASIC -e SELECT -d db11g -k 2 -r "db11g1,db11g2,db11g3"
    

    Solution not supported by Oracle. Creating dependency of resource manually.

    # Create service without specifying flag "-k" srvctl will use default network 1
    srvctl add service -s db11g_inbound -m BASIC -e SELECT -d db11g  -r "db11g1,db11g2,db11g3"
    
    # Getting current START and STOP Dependencies.
    
    crsctl status resource ora.db11g.db11g_inbound.svc -f |grep -E 'START_DEPENDENCIES|STOP_DEPENDENCIES'
    START_DEPENDENCIES=
    hard(ora.db11g.db,type:ora.cluster_vip_net1.type)
    weak(type:ora.listener.type)
    pullup(type:ora.cluster_vip_net1.type)
    pullup:always(ora.db11g.db)
    
    STOP_DEPENDENCIES=hard(intermediate:ora.db11g.db,intermediate:type:ora.cluster_vip_net1.type)
    
    # Modifying START Dependencies adding type cluster_vip_net (cluster_vip_net2 and cluster_vip_net3)
    crsctl modify resource ora.db11g.db11g_inbound.svc -attr\
     "START_DEPENDENCIES='hard(\
    ora.db11g.db,\
    type:ora.cluster_vip_net1.type,\
    type:ora.cluster_vip_net2.type,\
    type:ora.cluster_vip_net3.type)\
    weak(type:ora.listener.type) \
    pullup(type:ora.cluster_vip_net1.type) \
    pullup(type:ora.cluster_vip_net2.type) \
    pullup(type:ora.cluster_vip_net3.type) \
    pullup:always(ora.db11g.db)'"
    
    # Modifying STOP Dependencies adding type cluster_vip_net (cluster_vip_net2 and cluster_vip_net3)
    crsctl modify resource ora.db11g.db11g_inbound.svc -attr\
    "STOP_DEPENDENCIES='hard(\
    intermediate:ora.db11g.db,\
    intermediate:type:ora.cluster_vip_net1.type,\
    intermediate:type:ora.cluster_vip_net2.type,\
    intermediate:type:ora.cluster_vip_net3.type)'"
    
    # P.S The service db11g_inbound will not start if network (VIP IP) 1,2 and 3 is not avaliable, and you cannot stop network 1,2 and 3 if sevice db11g_inbound is online.
    # Starting Service Created
    srvctl start service -d db11g -s db11g_inbound
    
    

    You must understand your environment and see if  is better to create a service for each network or create a service to all networks.
    Remember that the service created always will be available in all networks (listeners), but in network maintenance may affect services if you manually created dependency.
    Checking if Service was registered in all Listeners

    
    ############### Network 1 #########################
     lsnrctl status listener_scan1
    ..
    .
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.63)(PORT=1521)))
    ..
    .
    Service "db11g_inbound" has 3 instance(s).
      Instance "db11g1", status READY, has 1 handler(s) for this service...
      Instance "db11g2", status READY, has 1 handler(s) for this service...
      Instance "db11g3", status READY, has 1 handler(s) for this service...
    ..
    .
    
     lsnrctl status listener
    ..
    .
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.60)(PORT=1521)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.52)(PORT=1521)))
    ..
    .
    Service "db11g_inbound" has 1 instance(s).
      Instance "db11g2", status READY, has 2 handler(s) for this service...
    ..
    .
    ############### Network 2 #########################
    lsnrctl status listener_south
    ..
    .
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SOUTH)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=140.120.120.101)(PORT=1522)))
    ..
    .
    Service "db11g_inbound" has 3 instance(s).
      Instance "db11g1", status READY, has 1 handler(s) for this service...
      Instance "db11g2", status READY, has 2 handler(s) for this service...
      Instance "db11g3", status READY, has 1 handler(s) for this service...
    ..
    .
    ############### Network 3 #########################
    lsnrctl status listener_east
    ..
    .
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_EAST)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=154.120.120.101)(PORT=1523)))
    ..
    .
    Service "db11g_inbound" has 3 instance(s).
      Instance "db11g1", status READY, has 1 handler(s) for this service...
      Instance "db11g2", status READY, has 2 handler(s) for this service...
      Instance "db11g3", status READY, has 1 handler(s) for this service...
    ..
    .
    

    Use the connect string above to connect  your clients on RAC Database 11.2

    
    ########### Connection String to Network 1 #####################
    ### Using SCAN ###########
    db11g_inbound =
     (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = global-scan.north.com)(PORT = 1521))
      (LOAD_BALANCE = YES)
      (CONNECT_DATA =
       (SERVER = DEDICATED)
       (SERVICE_NAME = db11g_inbound)
       (FAILOVER_MODE =
       (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5)
       )
      )
    
    ### Or Using VIP ###########
    db11g_inbound =
     (DESCRIPTION =
        (ADDRESS_LIST =
          (ADDRESS = (PROTOCOL = TCP)(HOST = northora01-vip.north.com)(PORT = 1521))
          (ADDRESS = (PROTOCOL = TCP)(HOST = northora02-vip.north.com)(PORT = 1521))
          (ADDRESS = (PROTOCOL = TCP)(HOST = northora03-vip.north.com)(PORT = 1521))
        )
      (LOAD_BALANCE = YES)
      (CONNECT_DATA =
       (SERVER = DEDICATED)
       (SERVICE_NAME = db11g_inbound)
       (FAILOVER_MODE =
       (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5)
       )
      )
    
    ########### Connection String to Network 2 #####################
    db11g_inbound =
     (DESCRIPTION =
        (ADDRESS_LIST =
          (ADDRESS = (PROTOCOL = TCP)(HOST = southora01-vip.north.com)(PORT = 1522))
          (ADDRESS = (PROTOCOL = TCP)(HOST = southora02-vip.north.com)(PORT = 1522))
          (ADDRESS = (PROTOCOL = TCP)(HOST = southora03-vip.north.com)(PORT = 1522))
        )
      (LOAD_BALANCE = YES)
      (CONNECT_DATA =
       (SERVER = DEDICATED)
       (SERVICE_NAME = db11g_inbound)
       (FAILOVER_MODE =
       (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5)
       )
      )
    
    ########### Connection String to Network 3 #####################
    db11g_inbound =
     (DESCRIPTION =
        (ADDRESS_LIST =
          (ADDRESS = (PROTOCOL = TCP)(HOST = eastora01-vip.north.com)(PORT = 1523))
          (ADDRESS = (PROTOCOL = TCP)(HOST = eastora02-vip.north.com)(PORT = 1523))
          (ADDRESS = (PROTOCOL = TCP)(HOST = eastora03-vip.north.com)(PORT = 1523))
        )
      (LOAD_BALANCE = YES)
      (CONNECT_DATA =
       (SERVER = DEDICATED)
       (SERVICE_NAME = db11g_inbound)
       (FAILOVER_MODE =
       (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5)
       )
      )
    
    

    Useful docs about how configure multiple network an RAC env.

    Data Guard: Redo Transport Services – How to use a separate network in a RAC environment. [ID 1210153.1] (11.1 or above)

    How to Configure A Second Listener on a Separate Network in 11.2 Grid Infrastructure [ID 1063571.1] (11.2)

    Data Guard Physical Standby 11.2 RAC Primary to RAC Standby using a second network [ID 1349977.1] (11.2)

    Hope this helps.

    Enjoy