Explaining: How to store OCR, Voting disks and ASM SPFILE on ASM Diskgroup (RAC or RAC Extended)

In 2011 I saw many doubts and concerns about how to store Voting,OCR and ASM SPFILE on ASM Diskgroup in this post I’ll show you how to set up your environment by applying best practices based on my experience.

To start I need explain some concepts:

Voting Disks is like an “Database Instance” and OCR is like a “Database”. During startup of Cluster Oracle first read and open all Voting Disks and after ASM be Started Oracle read and open all OCR.

So, Oracle does not need of ASM Instance  be started or  DISKGROUP be mounted to read and open Voting Disk.

Voting disks:

Voting Disk also known as Voting files: Is a file that manages information about node membership.

How they are stored in ASM?
Voting disks are placed directly on ASMDISK. Oracle Clusterware will store the votedisk on the disk within a disk group that holds the Voting Files. Oracle Clusterware does not rely on ASM to access the Voting Files, that’s means wich Oracle Clusterware does not need of Diskgroup to read and write on ASMDISK.
You cannot find/list Voting files using SQLPLUS(v$asm_diskgroup,v$asm_files,v$asm_alias), ASCMD utility or ASMCA gui.
You only know if exist a voting files in a ASMDISK (v$asm_disk using column VOTING_FILE). So, voting files not depend of Diskgroup to be accessed, does not mean that, we don’t need the diskgroup, diskgroup and voting file are linked by their settings.

Oracle Clusterware take configuration of DISKGROUP to configure the own voting files.
As Voting Disk are placed directly in ASMDISK of Diskgroup, we cannot use more than 1(one) Diskgroup.
The redundancy of voting files depend on ASMDISK not of Diskgroup. If you lose one ASMDISK  it’s means you lose one voting file. Differently when using files managed by Diskgroup.

  • When votedisk is on ASM diskgroup, no crsctl add option available. The number of votedisk is determined by the diskgroup redundancy. If more copy of votedisk is desired, one can move votedisk to a diskgroup with higher redundancy.
  • When votedisk is on ASM, no delete option available, one can only replace the existing votedisk group with another ASM diskgroup.

You cannot place Voting files in differents Diskgroup. To use a quorum failgroup is required if you are using RAC Extended or if you are using more than 1 Storage in your cluster.
The COMPATIBLE.ASM disk group compatibility attribute must be set to 11.2 or greater to store OCR or voting disk data in a disk group.

Oracle Cluster Registry (OCR) and ASM Server Parameter File (ASM SPFILE):

OCR: The Oracle RAC configuration information repository that manages information about the cluster node list and instance-to-node mapping information. OCR also manages information about Oracle Clusterware resource profiles for customized applications.

The OCR is totally different from Voting Disk. Oracle Clusterware rely on ASM to access the OCR and SPFILE. The OCR and SPFILE are stored similar to how Oracle Database files are stored. The extents are spread across all the disks in the diskgroup and the redundancy (which is at the extent level) is based on the redundancy of the disk group. For this reason you can only have one OCR in a diskgroup.

So, if your Diskgroup where OCR is stored become unavaliable you will lose your OCR and SPFILE. Then we need put OCR mirror in another disk group to support failure of disk group.

The interesting discussion is what happens if you have the OCR mirrored and one of the copies gets corrupt? You would expect that everything will continue to work seemlessly. Well.. Almost.. The real answer depends on when the corruption takes place.

If the corruption happens while the Oracle Clusterware stack is up and running, then the corruption will be tolerated and the Oracle Clusterware will continue to funtion without interruptions. Despite the corrupt copy. DBA is advised to repair this hardware/software problem that prevent OCR from accessing the device as soon as possible; alternatively, DBA can replace the failed device with another healthy device using the ocrconfig utility with -replace flag.

If however the corruption happens while the Oracle Clusterware stack is down, then it will not be possible to start it up until the failed device becomes online again or some administrative action using ocrconfig utility with -overwrite flag is taken.

Basic rules: You cannot create more than 1 (one) OCR or SPFILE in same Diskgroup.
The COMPATIBLE.ASM disk group compatibility attribute must be set to 11.2 or greater to store OCR or voting disk data in a disk group.

Best Practice for ASM is to have 2 diskgroups to store OCR.

Oracle Recommend: With Oracle Grid Infrastructure 11g Release 2, it is recommended to put the OCR and Voting Disks in Oracle ASM, using the same disk group you use for your database data.
I don’t agree!!!
I really don’t recommend put database files and clusterware files together. This can disrupt the management of the environment and cause dowtime. (e.g you never can stop this diskgroup)
Example:  The voting files are not stored in the diskgroup (+data), they are placed directly in asmdisk, then in case of maintenance in the diskgroup, for example to increase the size of Luns, you can not just remove the asmdisk, you must move the voting files to another place and achieve the maintenance in diskgroup.

Downtime ???? Yes… You can move only vote and ocr without downtime, but to move ASMSPIFILE you need downtime. This is required to ASM use new SPFILE and release the old Diskgroup. See: ORA-15027: active use of diskgroup precludes its dismount (With no database clients connected) [ID 1082876.1]

Voting files can be stored in only one diskgroup.
We can have X number of disk groups, during maintenance operations (replicate, drop, move,resize,clone etc.) the Clusterware files are unnecessarily involved.

So I recommend always create two small DISKGROUP:
+VOTE – Storing Voting files and OCR mirror
+CRS – Storing OCR and ASM Spfile.

Keep mind: You must make desing of LUNs of theses diskgroup (+VOTE, +CRS) before clusterware become avaliable to RAC databases (i.e Before install RAC).

Recomendation of Design of Luns:
Voting Disk need 300Mb
OCR and ASM SPFILE need 300M

Even using mirroring by Hardware (Storage), I recommend  create mirroring by ASM to Diskgroup that will store voting files, because these files will be configured as multiplexing.

Diskgroup VOTE:
Create 3 Luns of 500Mb each. If possible put each Lun in different controller, array or storage.

Diskgroup CRS:
If you are using mirror of storage or you are using only one storage, it’s recommended you create 1(one) Lun using external redundancy.

If you are not using mirror of storage or you are using more than one storage.

Using more than one storage :
1 Lun (500M) in each storage. Creating a diskgroup with normal redundancy.

If you are using one storage, but not using mirroring of storage.
Create 2 Luns of 500Mb each. Creating a diskgroup with normal redundancy.
Place each Lun in different controller, array or storage.

These Luns are exclusive to Cluster.

Returning to the old days…

So, we return to the old days (10.2), when we created separated luns to clusterware files using raw devices.

It may seem like a setback, but is a process that will facilitate management of environment, and makes it safer by separating files (clusterware files) that are extremely important  keeping the high availability cluster.

When you perform maintenance of the clusterware files you will change only  the diskgroup(CRS and VOTE) when you perform maintenance of the Diskgroup (Database Files or  ACFS)  the clusterware  files will not be involved.

Now, let’s do somes tests:

During Grid Install … What I can do to accomplish it?
We cannot achieve desired result during setup, but we can reconfigure it at end of installation. So, during install I always create a temporary diskgroup named +CRSTMP with external redundancy, asmdisk (lun) size 1G.

The diskgroup +CRSTMP will have one voting file, ocr and asm spfile.

Extremely important:  To perform the changes below all nodes in the cluster must be active. 

NEVER TRY REMOVE A ASMDISK  USING  ASMCA,ASMCMD or SQLPLUS if exists VOTING DISK on that ASMDISK.

You always must remove VOTING DISK from ASMDISK using ‘crsctl replace’  and after remove ASMDISK.

Checking if nodes is Actives:

$ olsnodes -s
lnxora01        Active
lnxora02        Active
lnxora03        Active

Use OCRCHECK to know where you OCR files are stored.

$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3848
         Available space (kbytes) :     258272
         ID                       : 1997055112
         Device/File Name         :    +CRSTMP
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

Use crsctl to know where Voting file is stored

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a0d6ea8dfb944fe7bfb799a451195a18 (ORCL:CRSTMP01) [CRSTMP]
Located 1 voting disk(s).

Use asmcmd to known where ASM SPFILE is stored

$ asmcmd spget
+CRSTMP/testcluster/ASMPARAMETERFILE/REGISTRY.253.772133609

Getting info about Voting Disk on ASM. We cannot see the voting file on ASM, we only know wich asmdisk he is stored.

SQL>
SET LINESIZE 150
COL PATH FOR A30
COL NAME FOR A10
COL HEADER_STATUS FOR A20
COL FAILGROUP FOR A20
COL FAILGROUP_TYPE FOR A20
COL VOTING_FILE FOR A20
SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE
FROM V$ASM_DISK
WHERE GROUP_NUMBER = ( SELECT GROUP_NUMBER
			 FROM V$ASM_DISKGROUP
			 WHERE NAME='CRSTMP');

NAME       PATH                           HEADER_STATUS        FAILGROUP            FAILGROUP_TYPE       VOTING_FILE
---------- ------------------------------ -------------------- -------------------- -------------------- --------------------
CRSTMP01   ORCL:CRSTMP01                  MEMBER               CRSTMP01             REGULAR              Y

Getting full name of OCR and ASM SPFILE on ASM

olsnodes -c : show name of cluster

$ olsnodes -c
tstcluster

set linesize 100
col FILES_OF_CLUSTER for a60

select concat('+'||gname, sys_connect_by_path(aname, '/')) FILES_OF_CLUSTER
     from ( select b.name gname, a.parent_index pindex, a.name aname,
              a.reference_index rindex , a.system_created, a.alias_directory,
              c.type file_type
       from v$asm_alias a, v$asm_diskgroup b, v$asm_file c
       where a.group_number = b.group_number
             and a.group_number = c.group_number(+)
             and a.file_number = c.file_number(+)
             and a.file_incarnation = c.incarnation(+)
     ) WHERE file_type in ( 'ASMPARAMETERFILE','OCRFILE')
start with (mod(pindex, power(2, 24))) = 0
            and rindex in
                ( select a.reference_index
                  from v$asm_alias a, v$asm_diskgroup b
                  where a.group_number = b.group_number
                        and (mod(a.parent_index, power(2, 24))) = 0
                        and a.name = LOWER('&CLUSTERNAME')
                )
connect by prior rindex = pindex;

Enter value for clustername: tstcluster
old  17:                         and a.name = LOWER('&CLUSTERNAME')
new  17:                         and a.name = LOWER('tstcluster')

FILES_OF_CLUSTER
---------------------------------------------------------
+CRSTMP/tstcluster/OCRFILE/REGISTRY.255.772133361
+CRSTMP/tstclsuter/ASMPARAMETERFILE/REGISTRY.253.772133609

After the disks are available on all hosts, we can start.

CRS01 and CRS02  will be used to diskgroup CRS

VOTE01,VOTE02 and VOTE03 will be used to diskgroup VOTE

 col path for a30
 col name for a20
 col header_status for a20
 select path,name,header_status from v$asm_disk
 where path like '%CRS%' or path like '%VOTE%';

PATH                           NAME                 HEADER_STATUS
------------------------------ -------------------- --------------------
ORCL:CRS01                                          PROVISIONED
ORCL:CRS02                                          PROVISIONED
ORCL:VOTE01                                         PROVISIONED
ORCL:VOTE02                                         PROVISIONED
ORCL:VOTE03                                         PROVISIONED
ORCL:CRSTMP01                  CRSTMP01             MEMBER

Creating Diskgroup VOTE each disk must be in different failgroup. I don’t add QUORUM failgroup because theses luns are on Storage. I recommend use QUORUM failgroup when you are placing disk out of your  environment. (e.g use NFS file-disk to quorum purpouses), because this disks cannot contain data.

SQL>
CREATE DISKGROUP VOTE NORMAL REDUNDANCY
     FAILGROUP STG1_C1 DISK 'ORCL:VOTE01'
     FAILGROUP STG1_C2 DISK 'ORCL:VOTE02'
     FAILGROUP STG1_C1_1 DISK 'ORCL:VOTE03'
     ATTRIBUTE 'compatible.asm' = '11.2.0.0.0';

Diskgroup created.

# starting diskgroup on others nodes
SQL> ! srvctl start diskgroup -g vote -n lnxora02,lnxora03

# checking if diskgroup is active on all nodes
SQL> ! srvctl status diskgroup -g vote
Disk Group vote is running on lnxora01,lnxora02,lnxora03

Creating DISKGROUP CRS:

SQL>
CREATE DISKGROUP CRS NORMAL REDUNDANCY
     FAILGROUP STG1_C1 DISK 'ORCL:CRS01'
     FAILGROUP STG1_C2 DISK 'ORCL:CRS02'
     ATTRIBUTE 'compatible.asm' = '11.2.0.0.0';

Diskgroup created.

# starting diskgroup on others nodes
SQL> ! srvctl start diskgroup -g crs -n lnxora02,lnxora03

# checking if diskgroup is active on all nodes
SQL> ! srvctl status diskgroup -g crs
Disk Group crs is running on lnxora01,lnxora02,lnxora03
SQL>
SET LINESIZE 150
COL PATH FOR A30
COL NAME FOR A10
COL HEADER_STATUS FOR A20
COL FAILGROUP FOR A20
COL FAILGROUP_TYPE FOR A20
COL VOTING_FILE FOR A20
SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE
FROM V$ASM_DISK
WHERE GROUP_NUMBER IN ( SELECT GROUP_NUMBER
			 FROM V$ASM_DISKGROUP
			 WHERE NAME IN ('CRS','VOTE'));

NAME       PATH                           HEADER_STATUS        FAILGROUP            FAILGROUP_TYPE       VOTING_FILE
---------- ------------------------------ -------------------- -------------------- -------------------- --------------------
VOTE03     ORCL:VOTE03                    MEMBER               STG1_C1_1            REGULAR              N
VOTE02     ORCL:VOTE02                    MEMBER               STG1_C2              REGULAR              N
VOTE01     ORCL:VOTE01                    MEMBER               STG1_C1              REGULAR              N
CRS01      ORCL:CRS01                     MEMBER               STG1_C1              REGULAR              N
CRS02      ORCL:CRS02                     MEMBER               STG1_C2              REGULAR              N

Moving Voting Files from +CRSTMP to +VOTE

$ crsctl replace votedisk +VOTE
Successful addition of voting disk aaa75b9e7ce24f39bfd9eecb3e3c0e38.
Successful addition of voting disk 873d51346cd34fc2bf9caa94999c4cd8.
Successful addition of voting disk acda8619b74c4fe8bf886ee6c9fe8d1a.
Successful deletion of voting disk a0d6ea8dfb944fe7bfb799a451195a18.
Successfully replaced voting disk group with +VOTE.
CRS-4266: Voting file(s) successfully replaced

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   aaa75b9e7ce24f39bfd9eecb3e3c0e38 (ORCL:VOTE01) [VOTE]
 2. ONLINE   873d51346cd34fc2bf9caa94999c4cd8 (ORCL:VOTE02) [VOTE]
 3. ONLINE   acda8619b74c4fe8bf886ee6c9fe8d1a (ORCL:VOTE03) [VOTE]
Located 3 voting disk(s).

SET LINESIZE 150
COL PATH FOR A30
COL NAME FOR A10
COL HEADER_STATUS FOR A20
COL FAILGROUP FOR A20
COL FAILGROUP_TYPE FOR A20
COL VOTING_FILE FOR A20
SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE
FROM V$ASM_DISK
WHERE GROUP_NUMBER = ( SELECT GROUP_NUMBER
			 FROM V$ASM_DISKGROUP
			 WHERE NAME='VOTE');

NAME       PATH                           HEADER_STATUS        FAILGROUP            FAILGROUP_TYPE       VOTING_FILE
---------- ------------------------------ -------------------- -------------------- -------------------- --------------------
VOTE03     ORCL:VOTE03                    MEMBER               STG1_C1_1            REGULAR              Y
VOTE02     ORCL:VOTE02                    MEMBER               STG1_C2              REGULAR              Y
VOTE01     ORCL:VOTE01                    MEMBER               STG1_C1              REGULAR              Y

Moving OCR to diskgroup +CRS and +VOTE and removing from diskgroup +CRSTMP

What is OCR determines whether the principal or mirror, is the order wich we add new OCR.
Therefore, we add first on the diskgroup CRS +  and later in diskgroup VOTE,  at time to remove the OCR on diskgroup CRSTMP the OCR on diskgroup CRS will become the principal.

# /u01/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3868
         Available space (kbytes) :     258252
         ID                       : 1997055112
         Device/File Name         :    +CRSTMP
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

# /u01/app/11.2.0/grid/bin/ocrconfig -add +CRS

# /u01/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3836
         Available space (kbytes) :     258284
         ID                       : 1997055112
         Device/File Name         :    +CRSTMP
                                    Device/File integrity check succeeded
         Device/File Name         :       +CRS
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

# /u01/app/11.2.0/grid/bin/ocrconfig -add +VOTE

 /u01/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3836
         Available space (kbytes) :     258284
         ID                       : 1997055112
         Device/File Name         :    +CRSTMP
                                    Device/File integrity check succeeded
         Device/File Name         :       +CRS
                                    Device/File integrity check succeeded
         Device/File Name         :      +VOTE
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

# /u01/app/11.2.0/grid/bin/ocrconfig -delete +CRSTMP

/u01/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3836
         Available space (kbytes) :     258284
         ID                       : 1997055112
         Device/File Name         :       +CRS
                                    Device/File integrity check succeeded
         Device/File Name         :      +VOTE
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

Moving ASM SPFILE to diskgroup +CRS

You will get the error that the file is still being used, but actually the file is copied to the file system and the profile is updated.

$ asmcmd spget
+CRSTMP/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772133609

$ asmcmd spmove '+CRSTMP/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772133609' '+CRS/tstcluster/spfileASM.ora'
ORA-15032: not all alterations performed
ORA-15028: ASM file '+CRSTMP/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772133609' not dropped; currently being accessed (DBD ERROR: OCIStmtExecute)

# checking if  file was copied and profile updated
$ asmcmd spget
+CRS/tstcluster/spfileASM.ora

Checking files of cluster on ASM

set linesize 100
col FILES_OF_CLUSTER for a60

select concat('+'||gname, sys_connect_by_path(aname, '/')) FILES_OF_CLUSTER
     from ( select b.name gname, a.parent_index pindex, a.name aname,
              a.reference_index rindex , a.system_created, a.alias_directory,
              c.type file_type
       from v$asm_alias a, v$asm_diskgroup b, v$asm_file c
       where a.group_number = b.group_number
             and a.group_number = c.group_number(+)
             and a.file_number = c.file_number(+)
             and a.file_incarnation = c.incarnation(+)
     ) WHERE file_type in ( 'ASMPARAMETERFILE','OCRFILE')
start with (mod(pindex, power(2, 24))) = 0
            and rindex in
                ( select a.reference_index
                  from v$asm_alias a, v$asm_diskgroup b
                  where a.group_number = b.group_number
                        and (mod(a.parent_index, power(2, 24))) = 0
                        and a.name = LOWER('&CLUSTERNAME')
                )
connect by prior rindex = pindex;
Enter value for clustername: tstcluster
old  17:                         and a.name = LOWER('&CLUSTERNAME')
new  17:                         and a.name = LOWER('tstcluster')

FILES_OF_CLUSTER
------------------------------------------------------------
+CRSTMP/tstcluster/OCRFILE/REGISTRY.255.772133361
+CRSTMP/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772133609
+VOTE/tstcluster/OCRFILE/REGISTRY.255.772207785
+CRS/tstcluster/OCRFILE/REGISTRY.255.772207425
+CRS/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772208263
+CRS/tstcluster/spfileASM.ora

In order the ASM can use the new SPFILE and  disconnect from the diskgroup + CRSTMP, we need to restart the cluster.

# /u01/app/11.2.0/grid/bin/crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'lnxora01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'lnxora01'
.
.
.
CRS-2673: Attempting to stop 'ora.crsd' on 'lnxora02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'lnxora02'
.
.
.
CRS-2673: Attempting to stop 'ora.crsd' on 'lnxora03'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'lnxora03'
.
.
.
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'lnxora01' has completed
.
.
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'lnxora02' has completed
.
.
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'lnxora03' has completed

# /u01/app/11.2.0/grid/bin/crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lnxora01'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lnxora02'
.

$ asmcmd spget
+CRS/tstcluster/spfileASM.ora

Now we can drop diskgroup +CRSTMP

SQL> ! srvctl stop diskgroup -g crstmp -n lnxora02,lnxora02

SQL> drop diskgroup crstmp including contents;

Diskgroup dropped.

SQL>
FILES_OF_CLUSTER
------------------------------------------------------------
+CRS/tstcluster/OCRFILE/REGISTRY.255.772207425
+CRS/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772211229
+CRS/tstcluster/spfileASM.ora
+VOTE/tstcluster/OCRFILE/REGISTRY.255.772207785
Adding a 3rd Voting File on NFS to a Cluster using Oracle ASM

In this post I’ll show how configure it on Linux, more detailed step, or how configure it in others platform you can use this Oracle white paper  (http://www.oracle.com/technetwork/database/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf)

Based on the above settings I’ll show you how easy it is to add a votedisk using ASM. (Linux only)

Preparing NFS Server: ( Oracle recommend use a exclusive host to 3rd votedisk)


# mkdir /votedisk
# vi /etc/exports
/votedisk *(rw,sync,all_squash,anonuid=54321,anongid=54325)

Setting Up NFS Clients
This conf above must be in all nodes of cluster

# cat /etc/filesystem
lnxnfs:/votedisk      /voting_disk    nfs     rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600       0       0

Mount the /voting_disk in all nodes of cluster and check if they are with right options

# mount /voting_disk

$ mount |grep voting_disk
lnxnfs:/votedisk on /voting_disk type nfs (rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,noac,addr=192.168.217.45)

Create a Disk-File to be used by ASM

$ dd if=/dev/zero of=/voting_disk/asm_vote_quorum bs=10M count=58
58+0 records in
58+0 records out
608174080 bytes (608 MB) copied, 3.68873 seconds, 165 MB/s

# chmod 660 /voting_disk/asm_vote_quorum
# chown oracle.asmadmin /voting_disk/asm_vote_quorum

# ls -ltr /voting_disk/asm_vote_quorum
-rw-rw---- 1 oracle asmadmin 608174080 Jan 10 20:00 /voting_disk/asm_vote_quorum

Adding the new Diskstring on ASM

SQL> show parameter asm_diskstring
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring                       string      ORCL:*

SQL> ALTER SYSTEM SET asm_diskstring ='ORCL:*','/voting_disk/asm_vote_quorum' SCOPE=both SID='*';

SQL> show parameter asm_diskstring
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------------
asm_diskstring                       string      ORCL:*, /voting_disk/asm_vote_quorum</pre>

$ asmcmd dsget
parameter:ORCL:*, /voting_disk/asm_vote_quorum
profile:ORCL:*,/voting_disk/asm_vote_quorum

Checking if this new disk is avaliable on ASM

$ kfod disks=all
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group
================================================================================
   1:        580 Mb /voting_disk/asm_vote_quorum             oracle   asmadmin
   2:        486 Mb ORCL:CRS01
   3:        486 Mb ORCL:CRS02
   .
   .
   .
  9:         580 Mb ORCL:VOTE01
  10:        580 Mb ORCL:VOTE02
  11:        580 Mb ORCL:VOTE03
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
     +ASM3 /u01/app/11.2.0/grid
     +ASM1 /u01/app/11.2.0/grid
     +ASM2 /u01/app/11.2.0/grid

SQL> col path for a30
SQL>
select path,header_status
from v$asm_disk
 where path like '%vote_quorum%';

PATH                           HEADER_STATUS
------------------------------ --------------------
/voting_disk/asm_vote_quorum   CANDIDATE

SQL>  ALTER DISKGROUP VOTE
      ADD
      QUORUM FAILGROUP STG_NFS DISK '/voting_disk/asm_vote_quorum';

Diskgroup altered.

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   aaa75b9e7ce24f39bfd9eecb3e3c0e38 (ORCL:VOTE01) [VOTE]
 2. ONLINE   873d51346cd34fc2bf9caa94999c4cd8 (ORCL:VOTE02) [VOTE]
 3. ONLINE   51f29389684e4f60bfb4b1683db8bd09 (/voting_disk/asm_vote_quorum) [VOTE]
Located 3 voting disk(s).

SQL>
SET LINESIZE 150
COL PATH FOR A30
COL NAME FOR A10
COL HEADER_STATUS FOR A20
COL FAILGROUP FOR A20
COL FAILGROUP_TYPE FOR A20
COL VOTING_FILE FOR A20
SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE
FROM V$ASM_DISK
WHERE GROUP_NUMBER = ( SELECT GROUP_NUMBER
			 FROM V$ASM_DISKGROUP
			 WHERE NAME='VOTE');

NAME       PATH                           HEADER_STATUS        FAILGROUP            FAILGROUP_TYPE       VOTING_FILE
---------- ------------------------------ -------------------- -------------------- -------------------- --------------------
VOTE01     ORCL:VOTE01                    MEMBER               STG1_C1              REGULAR              Y
VOTE02     ORCL:VOTE02                    MEMBER               STG1_C2              REGULAR              Y
VOTE03     ORCL:VOTE03                    MEMBER               STG1_C1_1            REGULAR              N
VOTE_0003  /voting_disk/asm_vote_quorum   MEMBER               STG_NFS              QUORUM               Y

### Use WAIT option to make sure wich you can remove asmdisk, it will not release the prompt until the rebalance operation completed.
SQL>  ALTER DISKGROUP VOTE
     DROP DISK 'VOTE03'
     REBALANCE POWER 3 WAIT;

Diskgroup altered.

NAME       PATH                           HEADER_STATUS        FAILGROUP            FAILGROUP_TYPE       VOTING_FILE
---------- ------------------------------ -------------------- -------------------- -------------------- --------------------
VOTE01     ORCL:VOTE01                    MEMBER               STG1_C1              REGULAR              Y
VOTE02     ORCL:VOTE02                    MEMBER               STG1_C2              REGULAR              Y
VOTE_0003  /voting_disk/asm_vote_quorum   MEMBER               STG_NFS              QUORUM               Y

 

Can we have 15 Voting Disk on ASM?

No. 15 voting files is allowed if you not storing voting on ASM. If you are using ASM the maximum number of voting files is 5. Because Oracle will take configuration of Diskgroup.
Using high number of voting disks can be useful when you have a big cluster environment with (e.g) 5 Storage Subsystem and 20 Hosts in a single Cluster. You must set up a voting file in each storage … but if you’re using only one storage voting 3 files is enough.

https://forums.oracle.com/forums/thread.jspa?messageID=10070225

Oracle Doc’s: You should have at least three voting disks, unless you have a storage device, such as a disk array, that provides external redundancy. Oracle recommends that you do not use more than 5 voting disks. The maximum number of voting disks that is supported is 15.
http://docs.oracle.com/cd/E11882_01/rac.112/e16794/crsref.htm#CHEJDHFH

See this example;

I configured 7 ASM DISK but ORACLE used only 5 ASM DISK.

SQL> CREATE DISKGROUP DG_VOTE HIGH REDUNDANCY
     FAILGROUP STG1 DISK 'ORCL:DG_VOTE01'
     FAILGROUP STG2 DISK 'ORCL:DG_VOTE02'
     FAILGROUP STG3 DISK 'ORCL:DG_VOTE03'
     FAILGROUP STG4 DISK 'ORCL:DG_VOTE04'
     FAILGROUP STG5 DISK 'ORCL:DG_VOTE05'
     FAILGROUP STG6 DISK 'ORCL:DG_VOTE06'
     FAILGROUP STG7 DISK 'ORCL:DG_VOTE07'
   ATTRIBUTE 'compatible.asm' = '11.2.0.0.0';

Diskgroup created.

SQL> ! srvctl start diskgroup -g DG_VOTE -n lnxora02,lnxora03

$  crsctl replace votedisk +DG_VOTE
CRS-4256: Updating the profile
Successful addition of voting disk 427f38b47ff24f52bf1228978354f1b2.
Successful addition of voting disk 891c4a40caed4f05bfac445b2fef2e14.
Successful addition of voting disk 5421865636524f5abf008becb19efe0e.
Successful addition of voting disk a803232576a44f1bbff65ab626f51c9e.
Successful addition of voting disk 346142ea30574f93bf870a117bea1a39.
Successful deletion of voting disk 2166953a27a14fcbbf38dae2c4049fa2.
Successfully replaced voting disk group with +DG_VOTE.

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   427f38b47ff24f52bf1228978354f1b2 (ORCL:DG_VOTE01) [DG_VOTE]
 2. ONLINE   891c4a40caed4f05bfac445b2fef2e14 (ORCL:DG_VOTE02) [DG_VOTE]
 3. ONLINE   5421865636524f5abf008becb19efe0e (ORCL:DG_VOTE03) [DG_VOTE]
 4. ONLINE   a803232576a44f1bbff65ab626f51c9e (ORCL:DG_VOTE04) [DG_VOTE]
 5. ONLINE   346142ea30574f93bf870a117bea1a39 (ORCL:DG_VOTE05) [DG_VOTE]

SQL >
SET LINESIZE 150
COL PATH FOR A30
COL NAME FOR A10
COL HEADER_STATUS FOR A20
COL FAILGROUP FOR A20
COL FAILGROUP_TYPE FOR A20
COL VOTING_FILE FOR A20
SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE
FROM V$ASM_DISK
WHERE GROUP_NUMBER = ( SELECT GROUP_NUMBER
			 FROM V$ASM_DISKGROUP
			 WHERE NAME='DG_VOTE');

NAME       PATH                           HEADER_STATUS        FAILGROUP            FAILGROUP_TYPE       VOTING_FILE
---------- ------------------------------ -------------------- -------------------- -------------------- --------------------
DG_VOTE01  ORCL:DG_VOTE01                 MEMBER               STG1                 REGULAR              Y
DG_VOTE02  ORCL:DG_VOTE02                 MEMBER               STG2                 REGULAR              Y
DG_VOTE03  ORCL:DG_VOTE03                 MEMBER               STG3                 REGULAR              Y
DG_VOTE04  ORCL:DG_VOTE04                 MEMBER               STG4                 REGULAR              Y
DG_VOTE05  ORCL:DG_VOTE05                 MEMBER               STG5                 REGULAR              Y
DG_VOTE06  ORCL:DG_VOTE06                 MEMBER               STG6                 REGULAR              N
DG_VOTE07  ORCL:DG_VOTE07                 MEMBER               STG7                 REGULAR              N
Errors and Workaround

ASM removed VOTEDISK from wrong ASMDISK (failgroup)… How fix it?

You can not choose which ASMDISK the votedisk will be removed. This can be a problem.
It is easy to solve this problem.

Follow this steps:


## As you configured an NFS then you can move the votedisk to NFS.

$ crsctl replace votedisk '/voting_disk/vote_temp

#### So, you can drop desired ASMDISK and ADD new ASMDISK with QUORUM option.
#### It's recommended you have 3 Failgroup (one failgroup in each storage) and 3rd failgroup is a quorum on nfs.
#### After reconfigure ASM Diskgroup VOTE you can move votedisk on nfs to ASM.

$ crsctl replace votedisk +VOTE

### Everthing will work

After restart Cluster CRS is not Starting, how fix it?

Problem: After restart Cluster in all nodes the CRS is not starting in some nodes after change OCR Location.
The node with problem was not updated the OCR Location, so he can trying find old diskgroup.

I changed OCR from +CRSTMP to +CRS, +VOTE

You can solve it’s manually:
The error on log crsd.log is look like:

2012-01-10 14:39:26.144: [ CRSMAIN][4039143920] Initializing OCR
2012-01-10 14:39:26.145: [ CRSMAIN][1089243456] Policy Engine is not initialized yet!
[   CLWAL][4039143920]clsw_Initialize: OLR initlevel [70000]
2012-01-10 14:39:32.712: [  OCRRAW][4039143920]proprioo: for disk 0 (+CRSTMP), id match (0), total id sets, (0) need recover (0), my votes (0), total votes (0), commit_lsn (0), lsn (0)
2012-01-10 14:39:32.712: [  OCRRAW][4039143920]proprioo: my id set: (723563391, 1028247821, 0, 0, 0)
2012-01-10 14:39:32.712: [  OCRRAW][4039143920]proprioo: 1st set: (0, 0, 0, 0, 0)
2012-01-10 14:39:32.712: [  OCRRAW][4039143920]proprioo: 2nd set: (0, 0, 0, 0, 0)
2012-01-10 14:39:32.838: [  OCRRAW][4039143920]utiid:problem validating header for owner db phy_addr=0
2012-01-10 14:39:32.838: [  OCRRAW][4039143920]proprinit:problem reading the bootblock or superbloc 26

2012-01-10 14:39:33.565: [  OCRAPI][4039143920]a_init:16!: Backend init unsuccessful : [26]
2012-01-10 14:39:33.570: [  CRSOCR][4039143920] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage
2012-01-10 14:39:33.570: [  CRSOCR][4039143920][PANIC] OCR Context is NULL(File: caaocr.cpp, line: 145)

2012-01-10 14:39:33.570: [    CRSD][4039143920][PANIC] CRSD Exiting. OCR Failed
2012-01-10 14:39:33.571: [    CRSD][4039143920] Done.
[/sourcode]

We can get error in two phrases: 
1
2012-01-10 14:39:32.712: [  OCRRAW][4039143920]proprioo: for disk 0 (+CRSTMP), id match (0), total id sets, (0) need recover (0), my votes (0), total votes (0), commit_lsn (0), lsn (0)
2012-01-10 14:39:33.570: [  CRSOCR][4039143920] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage

To solve it:

Connect on server wich CRS is working.

And see the content of file “cat /etc/oracle/ocr.loc”

In my case:

On node where CRS is working:

host: lnxora01
$ cat /etc/oracle/ocr.loc
#Device/file +CRSTMP getting replaced by device +CRS
ocrconfig_loc=+CRS
ocrmirrorconfig_loc=+VOTE
local_only=false

On node where CRS is not working:

host: lnxora02
$ cat /etc/oracle/ocr.loc
ocrconfig_loc=+CRSTMP
local_only=false

The file “/etc/oracle/ocr.loc” must be equal in all node, so I updated the ocr.loc on server with problem and all the CRS started without error

I need to do a few revisions in this post. (Sorry for grammar errors)

Enjoy…

.

About these ads

14 Comments on “Explaining: How to store OCR, Voting disks and ASM SPFILE on ASM Diskgroup (RAC or RAC Extended)”

  1. really helpful for those who needs to clear about asm and vd…
    i planned to do oracle10g RAC implementation in my laptop using oracle virtualbox..can you please guide me to finish it successfully..

  2. [...] 0 Times in 0 Posts Hi, Also you can try it: Explaining: How to store OCR, Voting disks and ASM SPFILE on ASM Diskgroup (RAC or RAC Extended) Due a lot of these question here I created this postFar as I can understand the purpose of QUORUM [...]

  3. [...] parameters of configuration. About how store Clusterware files on ASM I recommend you read it: Explaining: How to store OCR, Voting disks and ASM SPFILE on ASM Diskgroup (RAC or RAC Extended) Regards,No rebalance is done only you add more disks, So, the structures of storage must be the [...]

  4. SP says:

    Excellent explanation .. Thanks for the Post.

  5. anil says:

    Excellent work by you, i follow your blog to resolve the issues.I am planning to install extended distance RAC 11gr2 on Solaris 10 on 4 node cluster,2 node at a remote location and the interconnect is connected with dark fiber.I need suggestions from you on how to use quorum group for voting disk and how can i test the load balancing on four nodes, interconnectivity,storage and what is the most crucial test to be conducted to check the extended distance cluster is good and ready to use it for production.I appreciate your reply soon on this it’s kind of urgent

    • Levi Pereira says:

      Hi,
      About how configure quorum voting disk … you must use this doc and follow it: http://www.oracle.com/technetwork/products/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf

      About Test of Load Balancing I suggest you use Swingbench tool (running heavy workload by stressing on your env) : http://dominicgiles.com/swingbench.html . This test will validate your env to support your application.

      About storage and network test : You need test all possible failover (e.g network failure, storage failure, disk failure, site quorum failure, and so on)
      Also I suggest you do test of failure with cluster having high load (using swingbench).

      OOpen a SR to Oracle assist you on validation of your environment.

      Read FAQ : RAC: Frequently Asked Questions [ID 220970.1]
      Streched/Extended RAC — No Sub Category

      Hope this help,
      Levi Pereira

  6. Ken Hui says:

    Excellent writing. Answered all my questions on voting disks and OCR files. thank you.

  7. Pankaj Gupta says:

    Excellent explanation .. Thanks for the Post.

    -Pankaj Gupta

  8. Lemuel says:

    Greetings! Very helpful advice in this particular article!
    It’s the little changes which will make the greatest changes.
    Thanks for sharing!

  9. srinivas says:

    hi,

    I am using ASM 2 node environment
    please may i know the process of step how to start and how to stop (serial order)

  10. Mohammad says:

    Thanks a lot. I just copied what you wrote to my blog and referenced you their.

  11. Raju says:

    Thanks A Lot. Detailed Explanation.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 210 other followers