Explaining: How to store OCR, Voting disks and ASM SPFILE on ASM Diskgroup (RAC or RAC Extended)
Posted: 11/01/2012 | Author: Levi Pereira | Filed under: 11g R2, ASM - Automatic Storage Management, Clusterware / Grid Infrastructure, Linux, Operating System |19 CommentsIn 2011 I saw many doubts and concerns about how to store Voting,OCR and ASM SPFILE on ASM Diskgroup in this post I’ll show you how to set up your environment by applying best practices based on my experience.
To start I need explain some concepts:
Voting Disks is like an “Database Instance” and OCR is like a “Database”. During startup of Cluster Oracle first read and open all Voting Disks and after ASM be Started Oracle read and open all OCR.
So, Oracle does not need of ASM Instance be started or DISKGROUP be mounted to read and open Voting Disk.
Voting disks:
Voting Disk also known as Voting files: Is a file that manages information about node membership.
How they are stored in ASM?
Voting disks are placed directly on ASMDISK. Oracle Clusterware will store the votedisk on the disk within a disk group that holds the Voting Files. Oracle Clusterware does not rely on ASM to access the Voting Files, that’s means wich Oracle Clusterware does not need of Diskgroup to read and write on ASMDISK.
You cannot find/list Voting files using SQLPLUS(v$asm_diskgroup,v$asm_files,v$asm_alias), ASCMD utility or ASMCA gui.
You only know if exist a voting files in a ASMDISK (v$asm_disk using column VOTING_FILE). So, voting files not depend of Diskgroup to be accessed, does not mean that, we don’t need the diskgroup, diskgroup and voting file are linked by their settings.
Oracle Clusterware take configuration of DISKGROUP to configure the own voting files.
As Voting Disk are placed directly in ASMDISK of Diskgroup, we cannot use more than 1(one) Diskgroup.
The redundancy of voting files depend on ASMDISK not of Diskgroup. If you lose one ASMDISK it’s means you lose one voting file. Differently when using files managed by Diskgroup.
- When votedisk is on ASM diskgroup, no crsctl add option available. The number of votedisk is determined by the diskgroup redundancy. If more copy of votedisk is desired, one can move votedisk to a diskgroup with higher redundancy.
- When votedisk is on ASM, no delete option available, one can only replace the existing votedisk group with another ASM diskgroup.
You cannot place Voting files in differents Diskgroup. To use a quorum failgroup is required if you are using RAC Extended or if you are using more than 1 Storage in your cluster.
The COMPATIBLE.ASM disk group compatibility attribute must be set to 11.2 or greater to store OCR or voting disk data in a disk group.
Oracle Cluster Registry (OCR) and ASM Server Parameter File (ASM SPFILE):
OCR: The Oracle RAC configuration information repository that manages information about the cluster node list and instance-to-node mapping information. OCR also manages information about Oracle Clusterware resource profiles for customized applications.
The OCR is totally different from Voting Disk. Oracle Clusterware rely on ASM to access the OCR and SPFILE. The OCR and SPFILE are stored similar to how Oracle Database files are stored. The extents are spread across all the disks in the diskgroup and the redundancy (which is at the extent level) is based on the redundancy of the disk group. For this reason you can only have one OCR in a diskgroup.
So, if your Diskgroup where OCR is stored become unavaliable you will lose your OCR and SPFILE. Then we need put OCR mirror in another disk group to support failure of disk group.
The interesting discussion is what happens if you have the OCR mirrored and one of the copies gets corrupt? You would expect that everything will continue to work seemlessly. Well.. Almost.. The real answer depends on when the corruption takes place.
If the corruption happens while the Oracle Clusterware stack is up and running, then the corruption will be tolerated and the Oracle Clusterware will continue to funtion without interruptions. Despite the corrupt copy. DBA is advised to repair this hardware/software problem that prevent OCR from accessing the device as soon as possible; alternatively, DBA can replace the failed device with another healthy device using the ocrconfig utility with -replace flag.
If however the corruption happens while the Oracle Clusterware stack is down, then it will not be possible to start it up until the failed device becomes online again or some administrative action using ocrconfig utility with -overwrite flag is taken.
Basic rules: You cannot create more than 1 (one) OCR or SPFILE in same Diskgroup.
The COMPATIBLE.ASM disk group compatibility attribute must be set to 11.2 or greater to store OCR or voting disk data in a disk group.
Best Practice for ASM is to have 2 diskgroups to store OCR.
Oracle Recommend: With Oracle Grid Infrastructure 11g Release 2, it is recommended to put the OCR and Voting Disks in Oracle ASM, using the same disk group you use for your database data.
I don’t agree!!!
I really don’t recommend put database files and clusterware files together. This can disrupt the management of the environment and cause dowtime. (e.g you never can stop this diskgroup)
Example: The voting files are not stored in the diskgroup (+data), they are placed directly in asmdisk, then in case of maintenance in the diskgroup, for example to increase the size of Luns, you can not just remove the asmdisk, you must move the voting files to another place and achieve the maintenance in diskgroup.
Downtime ???? Yes… You can move only vote and ocr without downtime, but to move ASMSPIFILE you need downtime. This is required to ASM use new SPFILE and release the old Diskgroup. See: ORA-15027: active use of diskgroup precludes its dismount (With no database clients connected) [ID 1082876.1]
Voting files can be stored in only one diskgroup.
We can have X number of disk groups, during maintenance operations (replicate, drop, move,resize,clone etc.) the Clusterware files are unnecessarily involved.
So I recommend always create two small DISKGROUP:
+VOTE – Storing Voting files and OCR mirror
+CRS – Storing OCR and ASM Spfile.
Keep mind: You must make desing of LUNs of theses diskgroup (+VOTE, +CRS) before clusterware become avaliable to RAC databases (i.e Before install RAC).
Recomendation of Design of Luns:
Voting Disk need 300Mb
OCR and ASM SPFILE need 300M
Even using mirroring by Hardware (Storage), I recommend create mirroring by ASM to Diskgroup that will store voting files, because these files will be configured as multiplexing.
Diskgroup VOTE:
Create 3 Luns of 500Mb each. If possible put each Lun in different controller, array or storage.
Diskgroup CRS:
If you are using mirror of storage or you are using only one storage, it’s recommended you create 1(one) Lun using external redundancy.
If you are not using mirror of storage or you are using more than one storage.
Using more than one storage :
1 Lun (500M) in each storage. Creating a diskgroup with normal redundancy.
If you are using one storage, but not using mirroring of storage.
Create 2 Luns of 500Mb each. Creating a diskgroup with normal redundancy.
Place each Lun in different controller, array or storage.
These Luns are exclusive to Cluster.
Returning to the old days…
So, we return to the old days (10.2), when we created separated luns to clusterware files using raw devices.
It may seem like a setback, but is a process that will facilitate management of environment, and makes it safer by separating files (clusterware files) that are extremely important keeping the high availability cluster.
When you perform maintenance of the clusterware files you will change only the diskgroup(CRS and VOTE) when you perform maintenance of the Diskgroup (Database Files or ACFS) the clusterware files will not be involved.
Now, let’s do somes tests:
During Grid Install … What I can do to accomplish it?
We cannot achieve desired result during setup, but we can reconfigure it at end of installation. So, during install I always create a temporary diskgroup named +CRSTMP with external redundancy, asmdisk (lun) size 1G.
The diskgroup +CRSTMP will have one voting file, ocr and asm spfile.
Checking if nodes is Actives:
$ olsnodes -s lnxora01 Active lnxora02 Active lnxora03 Active
Use OCRCHECK to know where you OCR files are stored.
$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3848
Available space (kbytes) : 258272
ID : 1997055112
Device/File Name : +CRSTMP
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Use crsctl to know where Voting file is stored
$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE a0d6ea8dfb944fe7bfb799a451195a18 (ORCL:CRSTMP01) [CRSTMP] Located 1 voting disk(s).
Use asmcmd to known where ASM SPFILE is stored
$ asmcmd spget +CRSTMP/testcluster/ASMPARAMETERFILE/REGISTRY.253.772133609
Getting info about Voting Disk on ASM. We cannot see the voting file on ASM, we only know wich asmdisk he is stored.
SQL> SET LINESIZE 150 COL PATH FOR A30 COL NAME FOR A10 COL HEADER_STATUS FOR A20 COL FAILGROUP FOR A20 COL FAILGROUP_TYPE FOR A20 COL VOTING_FILE FOR A20 SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE FROM V$ASM_DISK WHERE GROUP_NUMBER = ( SELECT GROUP_NUMBER FROM V$ASM_DISKGROUP WHERE NAME='CRSTMP'); NAME PATH HEADER_STATUS FAILGROUP FAILGROUP_TYPE VOTING_FILE ---------- ------------------------------ -------------------- -------------------- -------------------- -------------------- CRSTMP01 ORCL:CRSTMP01 MEMBER CRSTMP01 REGULAR Y
Getting full name of OCR and ASM SPFILE on ASM
olsnodes -c : show name of cluster
$ olsnodes -c
tstcluster
set linesize 100
col FILES_OF_CLUSTER for a60
select concat('+'||gname, sys_connect_by_path(aname, '/')) FILES_OF_CLUSTER
from ( select b.name gname, a.parent_index pindex, a.name aname,
a.reference_index rindex , a.system_created, a.alias_directory,
c.type file_type
from v$asm_alias a, v$asm_diskgroup b, v$asm_file c
where a.group_number = b.group_number
and a.group_number = c.group_number(+)
and a.file_number = c.file_number(+)
and a.file_incarnation = c.incarnation(+)
) WHERE file_type in ( 'ASMPARAMETERFILE','OCRFILE')
start with (mod(pindex, power(2, 24))) = 0
and rindex in
( select a.reference_index
from v$asm_alias a, v$asm_diskgroup b
where a.group_number = b.group_number
and (mod(a.parent_index, power(2, 24))) = 0
and a.name = LOWER('&CLUSTERNAME')
)
connect by prior rindex = pindex;
Enter value for clustername: tstcluster
old 17: and a.name = LOWER('&CLUSTERNAME')
new 17: and a.name = LOWER('tstcluster')
FILES_OF_CLUSTER
---------------------------------------------------------
+CRSTMP/tstcluster/OCRFILE/REGISTRY.255.772133361
+CRSTMP/tstclsuter/ASMPARAMETERFILE/REGISTRY.253.772133609
After the disks are available on all hosts, we can start.
CRS01 and CRS02 will be used to diskgroup CRS
VOTE01,VOTE02 and VOTE03 will be used to diskgroup VOTE
col path for a30 col name for a20 col header_status for a20 select path,name,header_status from v$asm_disk where path like '%CRS%' or path like '%VOTE%'; PATH NAME HEADER_STATUS ------------------------------ -------------------- -------------------- ORCL:CRS01 PROVISIONED ORCL:CRS02 PROVISIONED ORCL:VOTE01 PROVISIONED ORCL:VOTE02 PROVISIONED ORCL:VOTE03 PROVISIONED ORCL:CRSTMP01 CRSTMP01 MEMBER
Creating Diskgroup VOTE each disk must be in different failgroup. I don’t add QUORUM failgroup because theses luns are on Storage. I recommend use QUORUM failgroup when you are placing disk out of your environment. (e.g use NFS file-disk to quorum purpouses), because this disks cannot contain data.
SQL>
CREATE DISKGROUP VOTE NORMAL REDUNDANCY
FAILGROUP STG1_C1 DISK 'ORCL:VOTE01'
FAILGROUP STG1_C2 DISK 'ORCL:VOTE02'
FAILGROUP STG1_C1_1 DISK 'ORCL:VOTE03'
ATTRIBUTE 'compatible.asm' = '11.2.0.0.0';
Diskgroup created.
# starting diskgroup on others nodes
SQL> ! srvctl start diskgroup -g vote -n lnxora02,lnxora03
# checking if diskgroup is active on all nodes
SQL> ! srvctl status diskgroup -g vote
Disk Group vote is running on lnxora01,lnxora02,lnxora03
Creating DISKGROUP CRS:
SQL>
CREATE DISKGROUP CRS NORMAL REDUNDANCY
FAILGROUP STG1_C1 DISK 'ORCL:CRS01'
FAILGROUP STG1_C2 DISK 'ORCL:CRS02'
ATTRIBUTE 'compatible.asm' = '11.2.0.0.0';
Diskgroup created.
# starting diskgroup on others nodes
SQL> ! srvctl start diskgroup -g crs -n lnxora02,lnxora03
# checking if diskgroup is active on all nodes
SQL> ! srvctl status diskgroup -g crs
Disk Group crs is running on lnxora01,lnxora02,lnxora03
SQL>
SET LINESIZE 150
COL PATH FOR A30
COL NAME FOR A10
COL HEADER_STATUS FOR A20
COL FAILGROUP FOR A20
COL FAILGROUP_TYPE FOR A20
COL VOTING_FILE FOR A20
SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE
FROM V$ASM_DISK
WHERE GROUP_NUMBER IN ( SELECT GROUP_NUMBER
FROM V$ASM_DISKGROUP
WHERE NAME IN ('CRS','VOTE'));
NAME PATH HEADER_STATUS FAILGROUP FAILGROUP_TYPE VOTING_FILE
---------- ------------------------------ -------------------- -------------------- -------------------- --------------------
VOTE03 ORCL:VOTE03 MEMBER STG1_C1_1 REGULAR N
VOTE02 ORCL:VOTE02 MEMBER STG1_C2 REGULAR N
VOTE01 ORCL:VOTE01 MEMBER STG1_C1 REGULAR N
CRS01 ORCL:CRS01 MEMBER STG1_C1 REGULAR N
CRS02 ORCL:CRS02 MEMBER STG1_C2 REGULAR N
Moving Voting Files from +CRSTMP to +VOTE
$ crsctl replace votedisk +VOTE Successful addition of voting disk aaa75b9e7ce24f39bfd9eecb3e3c0e38. Successful addition of voting disk 873d51346cd34fc2bf9caa94999c4cd8. Successful addition of voting disk acda8619b74c4fe8bf886ee6c9fe8d1a. Successful deletion of voting disk a0d6ea8dfb944fe7bfb799a451195a18. Successfully replaced voting disk group with +VOTE. CRS-4266: Voting file(s) successfully replaced $ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE aaa75b9e7ce24f39bfd9eecb3e3c0e38 (ORCL:VOTE01) [VOTE] 2. ONLINE 873d51346cd34fc2bf9caa94999c4cd8 (ORCL:VOTE02) [VOTE] 3. ONLINE acda8619b74c4fe8bf886ee6c9fe8d1a (ORCL:VOTE03) [VOTE] Located 3 voting disk(s). SET LINESIZE 150 COL PATH FOR A30 COL NAME FOR A10 COL HEADER_STATUS FOR A20 COL FAILGROUP FOR A20 COL FAILGROUP_TYPE FOR A20 COL VOTING_FILE FOR A20 SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE FROM V$ASM_DISK WHERE GROUP_NUMBER = ( SELECT GROUP_NUMBER FROM V$ASM_DISKGROUP WHERE NAME='VOTE'); NAME PATH HEADER_STATUS FAILGROUP FAILGROUP_TYPE VOTING_FILE ---------- ------------------------------ -------------------- -------------------- -------------------- -------------------- VOTE03 ORCL:VOTE03 MEMBER STG1_C1_1 REGULAR Y VOTE02 ORCL:VOTE02 MEMBER STG1_C2 REGULAR Y VOTE01 ORCL:VOTE01 MEMBER STG1_C1 REGULAR Y
Moving OCR to diskgroup +CRS and +VOTE and removing from diskgroup +CRSTMP
What is OCR determines whether the principal or mirror, is the order wich we add new OCR.
Therefore, we add first on the diskgroup CRS + and later in diskgroup VOTE, at time to remove the OCR on diskgroup CRSTMP the OCR on diskgroup CRS will become the principal.
# /u01/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3868
Available space (kbytes) : 258252
ID : 1997055112
Device/File Name : +CRSTMP
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
# /u01/app/11.2.0/grid/bin/ocrconfig -add +CRS
# /u01/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3836
Available space (kbytes) : 258284
ID : 1997055112
Device/File Name : +CRSTMP
Device/File integrity check succeeded
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
# /u01/app/11.2.0/grid/bin/ocrconfig -add +VOTE
/u01/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3836
Available space (kbytes) : 258284
ID : 1997055112
Device/File Name : +CRSTMP
Device/File integrity check succeeded
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File Name : +VOTE
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
# /u01/app/11.2.0/grid/bin/ocrconfig -delete +CRSTMP
/u01/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3836
Available space (kbytes) : 258284
ID : 1997055112
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File Name : +VOTE
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Moving ASM SPFILE to diskgroup +CRS
You will get the error that the file is still being used, but actually the file is copied to the file system and the profile is updated.
$ asmcmd spget +CRSTMP/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772133609 $ asmcmd spmove '+CRSTMP/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772133609' '+CRS/tstcluster/spfileASM.ora' ORA-15032: not all alterations performed ORA-15028: ASM file '+CRSTMP/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772133609' not dropped; currently being accessed (DBD ERROR: OCIStmtExecute) # checking if file was copied and profile updated $ asmcmd spget +CRS/tstcluster/spfileASM.ora
Checking files of cluster on ASM
set linesize 100
col FILES_OF_CLUSTER for a60
select concat('+'||gname, sys_connect_by_path(aname, '/')) FILES_OF_CLUSTER
from ( select b.name gname, a.parent_index pindex, a.name aname,
a.reference_index rindex , a.system_created, a.alias_directory,
c.type file_type
from v$asm_alias a, v$asm_diskgroup b, v$asm_file c
where a.group_number = b.group_number
and a.group_number = c.group_number(+)
and a.file_number = c.file_number(+)
and a.file_incarnation = c.incarnation(+)
) WHERE file_type in ( 'ASMPARAMETERFILE','OCRFILE')
start with (mod(pindex, power(2, 24))) = 0
and rindex in
( select a.reference_index
from v$asm_alias a, v$asm_diskgroup b
where a.group_number = b.group_number
and (mod(a.parent_index, power(2, 24))) = 0
and a.name = LOWER('&CLUSTERNAME')
)
connect by prior rindex = pindex;
Enter value for clustername: tstcluster
old 17: and a.name = LOWER('&CLUSTERNAME')
new 17: and a.name = LOWER('tstcluster')
FILES_OF_CLUSTER
------------------------------------------------------------
+CRSTMP/tstcluster/OCRFILE/REGISTRY.255.772133361
+CRSTMP/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772133609
+VOTE/tstcluster/OCRFILE/REGISTRY.255.772207785
+CRS/tstcluster/OCRFILE/REGISTRY.255.772207425
+CRS/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772208263
+CRS/tstcluster/spfileASM.ora
In order the ASM can use the new SPFILE and disconnect from the diskgroup + CRSTMP, we need to restart the cluster.
# /u01/app/11.2.0/grid/bin/crsctl stop cluster -all CRS-2673: Attempting to stop 'ora.crsd' on 'lnxora01' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'lnxora01' . . . CRS-2673: Attempting to stop 'ora.crsd' on 'lnxora02' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'lnxora02' . . . CRS-2673: Attempting to stop 'ora.crsd' on 'lnxora03' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'lnxora03' . . . CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'lnxora01' has completed . . CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'lnxora02' has completed . . CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'lnxora03' has completed # /u01/app/11.2.0/grid/bin/crsctl start cluster -all CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lnxora01' CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lnxora02' . $ asmcmd spget +CRS/tstcluster/spfileASM.ora
Now we can drop diskgroup +CRSTMP
SQL> ! srvctl stop diskgroup -g crstmp -n lnxora02,lnxora02 SQL> drop diskgroup crstmp including contents; Diskgroup dropped. SQL> FILES_OF_CLUSTER ------------------------------------------------------------ +CRS/tstcluster/OCRFILE/REGISTRY.255.772207425 +CRS/tstcluster/ASMPARAMETERFILE/REGISTRY.253.772211229 +CRS/tstcluster/spfileASM.ora +VOTE/tstcluster/OCRFILE/REGISTRY.255.772207785
Adding a 3rd Voting File on NFS to a Cluster using Oracle ASM
In this post I’ll show how configure it on Linux, more detailed step, or how configure it in others platform you can use this Oracle white paper (http://www.oracle.com/technetwork/database/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf)
Based on the above settings I’ll show you how easy it is to add a votedisk using ASM. (Linux only)
Preparing NFS Server: ( Oracle recommend use a exclusive host to 3rd votedisk)
# mkdir /votedisk # vi /etc/exports /votedisk *(rw,sync,all_squash,anonuid=54321,anongid=54325)
Setting Up NFS Clients
This conf above must be in all nodes of cluster
# cat /etc/filesystem lnxnfs:/votedisk /voting_disk nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600 0 0
Mount the /voting_disk in all nodes of cluster and check if they are with right options
# mount /voting_disk $ mount |grep voting_disk lnxnfs:/votedisk on /voting_disk type nfs (rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,noac,addr=192.168.217.45)
Create a Disk-File to be used by ASM
$ dd if=/dev/zero of=/voting_disk/asm_vote_quorum bs=10M count=58 58+0 records in 58+0 records out 608174080 bytes (608 MB) copied, 3.68873 seconds, 165 MB/s # chmod 660 /voting_disk/asm_vote_quorum # chown oracle.asmadmin /voting_disk/asm_vote_quorum # ls -ltr /voting_disk/asm_vote_quorum -rw-rw---- 1 oracle asmadmin 608174080 Jan 10 20:00 /voting_disk/asm_vote_quorum
Adding the new Diskstring on ASM
SQL> show parameter asm_diskstring NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_diskstring string ORCL:* SQL> ALTER SYSTEM SET asm_diskstring ='ORCL:*','/voting_disk/asm_vote_quorum' SCOPE=both SID='*'; SQL> show parameter asm_diskstring NAME TYPE VALUE ------------------------------------ ----------- ------------------------------------ asm_diskstring string ORCL:*, /voting_disk/asm_vote_quorum</pre> $ asmcmd dsget parameter:ORCL:*, /voting_disk/asm_vote_quorum profile:ORCL:*,/voting_disk/asm_vote_quorum
Checking if this new disk is avaliable on ASM
$ kfod disks=all
--------------------------------------------------------------------------------
Disk Size Path User Group
================================================================================
1: 580 Mb /voting_disk/asm_vote_quorum oracle asmadmin
2: 486 Mb ORCL:CRS01
3: 486 Mb ORCL:CRS02
.
.
.
9: 580 Mb ORCL:VOTE01
10: 580 Mb ORCL:VOTE02
11: 580 Mb ORCL:VOTE03
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
+ASM3 /u01/app/11.2.0/grid
+ASM1 /u01/app/11.2.0/grid
+ASM2 /u01/app/11.2.0/grid
SQL> col path for a30
SQL>
select path,header_status
from v$asm_disk
where path like '%vote_quorum%';
PATH HEADER_STATUS
------------------------------ --------------------
/voting_disk/asm_vote_quorum CANDIDATE
SQL> ALTER DISKGROUP VOTE
ADD
QUORUM FAILGROUP STG_NFS DISK '/voting_disk/asm_vote_quorum';
Diskgroup altered.
$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE aaa75b9e7ce24f39bfd9eecb3e3c0e38 (ORCL:VOTE01) [VOTE]
2. ONLINE 873d51346cd34fc2bf9caa94999c4cd8 (ORCL:VOTE02) [VOTE]
3. ONLINE 51f29389684e4f60bfb4b1683db8bd09 (/voting_disk/asm_vote_quorum) [VOTE]
Located 3 voting disk(s).
SQL>
SET LINESIZE 150
COL PATH FOR A30
COL NAME FOR A10
COL HEADER_STATUS FOR A20
COL FAILGROUP FOR A20
COL FAILGROUP_TYPE FOR A20
COL VOTING_FILE FOR A20
SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE
FROM V$ASM_DISK
WHERE GROUP_NUMBER = ( SELECT GROUP_NUMBER
FROM V$ASM_DISKGROUP
WHERE NAME='VOTE');
NAME PATH HEADER_STATUS FAILGROUP FAILGROUP_TYPE VOTING_FILE
---------- ------------------------------ -------------------- -------------------- -------------------- --------------------
VOTE01 ORCL:VOTE01 MEMBER STG1_C1 REGULAR Y
VOTE02 ORCL:VOTE02 MEMBER STG1_C2 REGULAR Y
VOTE03 ORCL:VOTE03 MEMBER STG1_C1_1 REGULAR N
VOTE_0003 /voting_disk/asm_vote_quorum MEMBER STG_NFS QUORUM Y
### Use WAIT option to make sure wich you can remove asmdisk, it will not release the prompt until the rebalance operation completed.
SQL> ALTER DISKGROUP VOTE
DROP DISK 'VOTE03'
REBALANCE POWER 3 WAIT;
Diskgroup altered.
NAME PATH HEADER_STATUS FAILGROUP FAILGROUP_TYPE VOTING_FILE
---------- ------------------------------ -------------------- -------------------- -------------------- --------------------
VOTE01 ORCL:VOTE01 MEMBER STG1_C1 REGULAR Y
VOTE02 ORCL:VOTE02 MEMBER STG1_C2 REGULAR Y
VOTE_0003 /voting_disk/asm_vote_quorum MEMBER STG_NFS QUORUM Y
Can we have 15 Voting Disk on ASM?
No. 15 voting files is allowed if you not storing voting on ASM. If you are using ASM the maximum number of voting files is 5. Because Oracle will take configuration of Diskgroup.
Using high number of voting disks can be useful when you have a big cluster environment with (e.g) 5 Storage Subsystem and 20 Hosts in a single Cluster. You must set up a voting file in each storage … but if you’re using only one storage voting 3 files is enough.
https://forums.oracle.com/forums/thread.jspa?messageID=10070225
Oracle Doc’s: You should have at least three voting disks, unless you have a storage device, such as a disk array, that provides external redundancy. Oracle recommends that you do not use more than 5 voting disks. The maximum number of voting disks that is supported is 15.
http://docs.oracle.com/cd/E11882_01/rac.112/e16794/crsref.htm#CHEJDHFH
See this example;
I configured 7 ASM DISK but ORACLE used only 5 ASM DISK.
SQL> CREATE DISKGROUP DG_VOTE HIGH REDUNDANCY
FAILGROUP STG1 DISK 'ORCL:DG_VOTE01'
FAILGROUP STG2 DISK 'ORCL:DG_VOTE02'
FAILGROUP STG3 DISK 'ORCL:DG_VOTE03'
FAILGROUP STG4 DISK 'ORCL:DG_VOTE04'
FAILGROUP STG5 DISK 'ORCL:DG_VOTE05'
FAILGROUP STG6 DISK 'ORCL:DG_VOTE06'
FAILGROUP STG7 DISK 'ORCL:DG_VOTE07'
ATTRIBUTE 'compatible.asm' = '11.2.0.0.0';
Diskgroup created.
SQL> ! srvctl start diskgroup -g DG_VOTE -n lnxora02,lnxora03
$ crsctl replace votedisk +DG_VOTE
CRS-4256: Updating the profile
Successful addition of voting disk 427f38b47ff24f52bf1228978354f1b2.
Successful addition of voting disk 891c4a40caed4f05bfac445b2fef2e14.
Successful addition of voting disk 5421865636524f5abf008becb19efe0e.
Successful addition of voting disk a803232576a44f1bbff65ab626f51c9e.
Successful addition of voting disk 346142ea30574f93bf870a117bea1a39.
Successful deletion of voting disk 2166953a27a14fcbbf38dae2c4049fa2.
Successfully replaced voting disk group with +DG_VOTE.
$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 427f38b47ff24f52bf1228978354f1b2 (ORCL:DG_VOTE01) [DG_VOTE]
2. ONLINE 891c4a40caed4f05bfac445b2fef2e14 (ORCL:DG_VOTE02) [DG_VOTE]
3. ONLINE 5421865636524f5abf008becb19efe0e (ORCL:DG_VOTE03) [DG_VOTE]
4. ONLINE a803232576a44f1bbff65ab626f51c9e (ORCL:DG_VOTE04) [DG_VOTE]
5. ONLINE 346142ea30574f93bf870a117bea1a39 (ORCL:DG_VOTE05) [DG_VOTE]
SQL >
SET LINESIZE 150
COL PATH FOR A30
COL NAME FOR A10
COL HEADER_STATUS FOR A20
COL FAILGROUP FOR A20
COL FAILGROUP_TYPE FOR A20
COL VOTING_FILE FOR A20
SELECT NAME,PATH,HEADER_STATUS,FAILGROUP, FAILGROUP_TYPE, VOTING_FILE
FROM V$ASM_DISK
WHERE GROUP_NUMBER = ( SELECT GROUP_NUMBER
FROM V$ASM_DISKGROUP
WHERE NAME='DG_VOTE');
NAME PATH HEADER_STATUS FAILGROUP FAILGROUP_TYPE VOTING_FILE
---------- ------------------------------ -------------------- -------------------- -------------------- --------------------
DG_VOTE01 ORCL:DG_VOTE01 MEMBER STG1 REGULAR Y
DG_VOTE02 ORCL:DG_VOTE02 MEMBER STG2 REGULAR Y
DG_VOTE03 ORCL:DG_VOTE03 MEMBER STG3 REGULAR Y
DG_VOTE04 ORCL:DG_VOTE04 MEMBER STG4 REGULAR Y
DG_VOTE05 ORCL:DG_VOTE05 MEMBER STG5 REGULAR Y
DG_VOTE06 ORCL:DG_VOTE06 MEMBER STG6 REGULAR N
DG_VOTE07 ORCL:DG_VOTE07 MEMBER STG7 REGULAR N
Errors and Workaround
ASM removed VOTEDISK from wrong ASMDISK (failgroup)… How fix it?
You can not choose which ASMDISK the votedisk will be removed. This can be a problem.
It is easy to solve this problem.
Follow this steps:
## As you configured an NFS then you can move the votedisk to NFS. $ crsctl replace votedisk '/voting_disk/vote_temp #### So, you can drop desired ASMDISK and ADD new ASMDISK with QUORUM option. #### It's recommended you have 3 Failgroup (one failgroup in each storage) and 3rd failgroup is a quorum on nfs. #### After reconfigure ASM Diskgroup VOTE you can move votedisk on nfs to ASM. $ crsctl replace votedisk +VOTE ### Everthing will work
After restart Cluster CRS is not Starting, how fix it?
Problem: After restart Cluster in all nodes the CRS is not starting in some nodes after change OCR Location.
The node with problem was not updated the OCR Location, so he can trying find old diskgroup.
I changed OCR from +CRSTMP to +CRS, +VOTE
You can solve it’s manually:
The error on log crsd.log is look like:
2012-01-10 14:39:26.144: [ CRSMAIN][4039143920] Initializing OCR 2012-01-10 14:39:26.145: [ CRSMAIN][1089243456] Policy Engine is not initialized yet! [ CLWAL][4039143920]clsw_Initialize: OLR initlevel [70000] 2012-01-10 14:39:32.712: [ OCRRAW][4039143920]proprioo: for disk 0 (+CRSTMP), id match (0), total id sets, (0) need recover (0), my votes (0), total votes (0), commit_lsn (0), lsn (0) 2012-01-10 14:39:32.712: [ OCRRAW][4039143920]proprioo: my id set: (723563391, 1028247821, 0, 0, 0) 2012-01-10 14:39:32.712: [ OCRRAW][4039143920]proprioo: 1st set: (0, 0, 0, 0, 0) 2012-01-10 14:39:32.712: [ OCRRAW][4039143920]proprioo: 2nd set: (0, 0, 0, 0, 0) 2012-01-10 14:39:32.838: [ OCRRAW][4039143920]utiid:problem validating header for owner db phy_addr=0 2012-01-10 14:39:32.838: [ OCRRAW][4039143920]proprinit:problem reading the bootblock or superbloc 26 2012-01-10 14:39:33.565: [ OCRAPI][4039143920]a_init:16!: Backend init unsuccessful : [26] 2012-01-10 14:39:33.570: [ CRSOCR][4039143920] OCR context init failure. Error: PROC-26: Error while accessing the physical storage 2012-01-10 14:39:33.570: [ CRSOCR][4039143920][PANIC] OCR Context is NULL(File: caaocr.cpp, line: 145) 2012-01-10 14:39:33.570: [ CRSD][4039143920][PANIC] CRSD Exiting. OCR Failed 2012-01-10 14:39:33.571: [ CRSD][4039143920] Done. [/sourcode] We can get error in two phrases: 2012-01-10 14:39:32.712: [ OCRRAW][4039143920]proprioo: for disk 0 (+CRSTMP), id match (0), total id sets, (0) need recover (0), my votes (0), total votes (0), commit_lsn (0), lsn (0) 2012-01-10 14:39:33.570: [ CRSOCR][4039143920] OCR context init failure. Error: PROC-26: Error while accessing the physical storage
To solve it:
Connect on server wich CRS is working.
And see the content of file “cat /etc/oracle/ocr.loc”
In my case:
On node where CRS is working:
host: lnxora01 $ cat /etc/oracle/ocr.loc #Device/file +CRSTMP getting replaced by device +CRS ocrconfig_loc=+CRS ocrmirrorconfig_loc=+VOTE local_only=false
On node where CRS is not working:
host: lnxora02 $ cat /etc/oracle/ocr.loc ocrconfig_loc=+CRSTMP local_only=false
The file “/etc/oracle/ocr.loc” must be equal in all node, so I updated the ocr.loc on server with problem and all the CRS started without error
I need to do a few revisions in this post. (Sorry for grammar errors)
Enjoy…
Rate this:
Share this:
- Click to share on Facebook (Opens in new window)
- Click to share on Twitter (Opens in new window)
- Click to share on LinkedIn (Opens in new window)
- Click to email a link to a friend (Opens in new window)
- Click to share on Reddit (Opens in new window)
- Click to share on Tumblr (Opens in new window)
- Click to share on Pinterest (Opens in new window)
- Click to print (Opens in new window)
Levi Pereira | Oracle Blog 
really helpful for those who needs to clear about asm and vd…
i planned to do oracle10g RAC implementation in my laptop using oracle virtualbox..can you please guide me to finish it successfully..
LikeLike
Very good and very detailed explanation of the matter. Congratulations! Very cool!
LikeLike
[…] 0 Times in 0 Posts Hi, Also you can try it: Explaining: How to store OCR, Voting disks and ASM SPFILE on ASM Diskgroup (RAC or RAC Extended) Due a lot of these question here I created this postFar as I can understand the purpose of QUORUM […]
LikeLike
[…] parameters of configuration. About how store Clusterware files on ASM I recommend you read it: Explaining: How to store OCR, Voting disks and ASM SPFILE on ASM Diskgroup (RAC or RAC Extended) Regards,No rebalance is done only you add more disks, So, the structures of storage must be the […]
LikeLike
Excellent explanation .. Thanks for the Post.
LikeLike
Excellent work by you, i follow your blog to resolve the issues.I am planning to install extended distance RAC 11gr2 on Solaris 10 on 4 node cluster,2 node at a remote location and the interconnect is connected with dark fiber.I need suggestions from you on how to use quorum group for voting disk and how can i test the load balancing on four nodes, interconnectivity,storage and what is the most crucial test to be conducted to check the extended distance cluster is good and ready to use it for production.I appreciate your reply soon on this it’s kind of urgent
LikeLike
Hi,
About how configure quorum voting disk … you must use this doc and follow it: http://www.oracle.com/technetwork/products/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf
About Test of Load Balancing I suggest you use Swingbench tool (running heavy workload by stressing on your env) : http://dominicgiles.com/swingbench.html . This test will validate your env to support your application.
About storage and network test : You need test all possible failover (e.g network failure, storage failure, disk failure, site quorum failure, and so on)
Also I suggest you do test of failure with cluster having high load (using swingbench).
OOpen a SR to Oracle assist you on validation of your environment.
Read FAQ : RAC: Frequently Asked Questions [ID 220970.1]
Streched/Extended RAC — No Sub Category
Hope this help,
Levi Pereira
LikeLike
Excellent writing. Answered all my questions on voting disks and OCR files. thank you.
LikeLike
Excellent explanation .. Thanks for the Post.
-Pankaj Gupta
LikeLike
Greetings! Very helpful advice in this particular article!
It’s the little changes which will make the greatest changes.
Thanks for sharing!
LikeLike
hi,
I am using ASM 2 node environment
please may i know the process of step how to start and how to stop (serial order)
LikeLike
Thanks a lot. I just copied what you wrote to my blog and referenced you their.
LikeLike
Nice one …!
LikeLike
Thanks A Lot. Detailed Explanation.
LikeLike
Sorry, I am having hard time understanding the sentences 😦 I can feel that this is great stuff. But couldn’t understand it properly. Seems like translation from some other language 😦
Please don’t take it as offense. I tried to read it 3-4times but couldn’t understand. Thank you very much for your time
LikeLike
Especially these…
Oracle Clusterware take configuration of DISKGROUP to configure the own voting files.
As Voting Disk are placed directly in ASMDISK of Diskgroup, we cannot use more than 1(one) Diskgroup. ==> What is the relation here?
The redundancy of voting files depend on ASMDISK not of Diskgroup. If you lose one ASMDISK it’s means you lose one voting file. Differently when using files managed by Diskgroup. ==> What do we mean by “redundancy of voting files depend on ASMDISK not of DISKGROUP”? Something is missing here
When votedisk is on ASM diskgroup, no crsctl add option available. The number of votedisk is determined by the diskgroup redundancy. ==> This is negating with the above statement where it says “redundancy of voting files depend on ASMDISK not of Diskgroup”
Thank you
LikeLike
[…] Ref:https://levipereira.wordpress.com/2012/01/11/explaining-how-to-store-ocr-voting-disks-and-asm-spfile… […]
LikeLike
I did the procedure step by step and it was finally successfully terminated. I couldn’t beleve it, but it was real. Oh my god!!
Thank you so much Levi for share your knowledge and experience.
God bless you!
LikeLike
I’m glad you succeed.
LikeLike