Oracle ACFS Filesystem managed by OHAS on Oracle Restart

Bad news from Oracle 11.2.0.2:

Oracle ACFS and Oracle Restart


Oracle Restart does not support root-based Oracle ACFS resources for this release. Consequently, the following operations are not automatically performed:

Loading Oracle ACFS drivers
Mounting Oracle ACFS file systems listed in the Oracle ACFS mount registry
Mounting resource-based Oracle ACFS database home file systems
The Oracle ACFS resources associated with these actions are not created for Oracle Restart configurations.

While Oracle ACFS resource management is fully supported for Oracle Grid Infrastructure configurations, the Oracle ACFS resource-based management actions must be replaced with alternative, sometimes manual, operations in Oracle Restart configurations.

http://download.oracle.com/docs/cd/E11882_01/server.112/e16102/asmfs_extra.htm#CACBDGCC

Big Question!
Why use ACFS (i.e Cluster Filesystem) on Single Node Installation? It makes no sense!!!
Using O.S filesystem I have all resources of ACFS (Performance, Manageability and Availability).
And using ACFS on Single Node (Oracle Restart) is very poor many task must be performed manually (like load drivers, mountfilesystem,etc) .
Particularly I do not use ACFS on Single Nodes, I see no need for that.

I created this post just for fun and to help those  wich use ACFS on Oracle Restart.

BUT…
ACFS has a feature that does not have on S.O filesystem.

ACFS Snapshots

ACFS provides snapshotting capability for the respective filesystem. This snapshot uses the First Copy-on-Write (FCOW) methodology to enable a consistent, version-based, online view of the source filesystem.
Snapshots are immutable views of the source file system as it appeared at a specific point in time.
Snapshots are initially a sparse filesystem, as the source filesystem files change, the before-image extent of that file is copied into the snapshot directory.
Accessing the snapshot will always provide a point-in-time view of a file, thus ACFS snapshots can be very useful for file-based recovery or for filesystem backups. If file level recovery is needed (for the base filesystem), it can be performed using standard file copy or replace commands.
A possible use case scenario for snapshots can be to create a consistent recovery point set between the database ORACLE_HOME and the database.
This is useful, for example, when a recovery point needs to be established before applying a database patchset.

Important: Be very careful using Oracle ACFS filesystem to ORACLE_HOME. When you create a database using this ORACLE_HOME this database is automatically registred on OHAS (Oracle Restart). If you restart Server OHAS will try start this database automatically, but the filesystem (ACFS) of ORACLE_HOME is not mounted automatically must be done manually , therefore all databases of this ORACLE_HOME must be started manually too.

I’ll show you how to automate the process of load and drivers of ACFS and mounting/umounting filesystem using OHAS.

LINUX PLATAFORM

Create a ACFS Filesystem.

I will create a ACFS Filesystem on DISKGROUP “DG_ORAHOME11GR2_1”

Create ACFS for Database Home

Configure Oracle Home location, Size (GB), user and group.

Creating ACFS…

ASMCA does not support root-based Oracle ACFS resources, so must be done manually.

Oracle automatically generate a script to mount this filesystem.

Volume created…and ENABLED

ACFS MOUNTED…..

If you restart your Server, the ASCS drivers will be not loaded automatically, therefore te TAB (Volumes and ASM Cluster File Systems) of ASMCA will be inactive.

Loading ACFS Drivers Automatically

To configure Autostart of ACFS drivers during node reboot on Linux

Add an acfsload start command in /etc/rc.local.so they are automatically loaded during node bootup

cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

/sbin/modprobe hangcheck-timer
/u01/app/grid/product/11.2.0/grid/bin/acfsload start –s
devlabel restart

Setting up  AUTO MOUNT/AUTO UMOUNT of Filesystem on OHAS. 

We can use OHAS to start, stop, monitor and restart applications. Using feature RESOURCE of OHAS.

Let’s create a script to mount/umount ACFS Filesystem.

Using Oracle user (oracle)

cd /u01/app/grid/product/11.2.0/grid/crs/script

vi acfs_ORAHOME11GR2.sh
#!/bin/sh
case $1 in
'start')
# Check if Volume is Enabled if not enable volume

if [ ! -f /dev/asm/db112_dbh1-220 ]; then
ORACLE_HOME=/u01/app/grid/product/11.2.0/grid
ORACLE_SID=+ASM
$ORACLE_HOME/bin/asmcmd <<EOF
volenable -G DG_ORAHOME11GR2_1 db112_dbh1
EOF
fi
# Mount filesystem

/usr/bin/sudo /bin/mount -t acfs /dev/asm/db112_dbh1-220 /u01/app/oracle/product/11.2.0/dbhome_1

# Change permission of Filesystem
 if [ $? = "0" ]; then
 /usr/bin/sudo  /bin/chown oracle:oinstall /u01/app/oracle/product/11.2.0/dbhome_1
 /usr/bin/sudo  /bin/chmod 775 /u01/app/oracle/product/11.2.0/dbhome_1
  exit 0
 fi
 RET=$?
;;

# Stop Filesystem
'stop')
/usr/bin/sudo /bin/umount -t acfs /dev/asm/db112_dbh1-220
 RET=$?
;;
'clean')
/usr/bin/sudo  /bin/umount -t acfs /dev/asm/db112_dbh1-220
  RET=$?
    ;;
# Check if Filesystem is Mounted
'check')
  OUTCMD=`/bin/mount |grep '/dev/asm/db112_dbh1-220' |wc -l`
  if [ $OUTCMD = 1 ]; then
  RET=0
  else
  RET=1
  fi
;;
esac
# 0: success; 1 : error
if [ $RET -eq 0 ]; then
exit 0
else
exit 1
fi

I used the “sudo” to perform root-tasks. So you must allow user oracle perform root actions without password.

Using root privileges (or root user)  type “visudo” to edit sudoers file.

# visudo

Comment the line "Defaults    requiretty"
# Defaults    requiretty

Add user Oracle above this:
# %wheel        ALL=(ALL)       NOPASSWD: ALL
oracle          ALL=(ALL)       NOPASSWD: /bin/chown, /bin/chmod, /bin/mount, /bin/umount

Registering Resource in OHAS

Register Resource “acfs.orahome11gR2_1.fs” on OHAS creating dependency of DISKGROUP “DG_ORAHOME11GR2_1”

START_DEPENDENCIES: Specifies a set of relationships that OHAS considers when starting a resource.
You can specify a space-separated list of dependencies on several resources and resource types on which a particular resource can depend.
hard: Specify a hard start dependency for a resource when you want the resource to start only when a particular resource or resource of a particular type starts.
Oracle recommends that resources with hard start dependencies also have pullup start dependencies.
pullup: When you specify the pullup start dependency for a resource, then this resource starts as a result of named resources starting.

So, We must specify the START_DEPENDENCIES referencing to DISKGROUP DG_ORAHOME11GR2_1 where are ORACLE_HOME filesystem, that’s means when you try start the resource “acfs.orahome11gR2_1.fs” the DISKGROUP DG_ORAHOME11GR2_1 must be started, if DISKGROUP DG_ORAHOME11GR2_1 is not started “pullup” will try start DISKGROUP DG_ORAHOME11GR2_1 before try start resource “acfs.orahome11gR2_1.fs”.

STOP_DEPENDENCIES: Specifies a set of relationships that OHAS considers when stopping a resource.
hard: Specify a hard stop dependency for a resource that you want to stop when named resources or resources of a particular resource type stop.

So, if we try to stop (using OHAS) DISKGROUP DG_ORAHOME11GR2_1 when ACFS (ORACLE_HOME) remain mounted (ONLINE) OHAS must raise CRS-2529

CRS-2529: Unable to act on  because that would require stopping or relocating ,
but the force option was not specified

 

crsctl add resource acfs.orahome11gR2_1.fs \
-type local_resource \
-attr "\
ACTION_SCRIPT=/u01/app/grid/product/11.2.0/grid/crs/script/acfs_ORAHOME11GR2.sh,\
AUTO_START=always,\
START_TIMEOUT=100,\
STOP_TIMEOUT=100,\
CHECK_INTERVAL=10,\
START_DEPENDENCIES=hard(ora.DG_ORAHOME11GR2_1.dg)pullup(ora.DG_ORAHOME11GR2_1.dg),\
STOP_DEPENDENCIES='hard(ora.DG_ORAHOME11GR2_1.dg)'"

More info about attributes used here you can found here:
http://download.oracle.com/docs/cd/E11882_01/rac.112/e16794/resatt.htm

$ crsctl status resource acfs.orahome11gR2_1.fs
NAME=acfs.orahome11gR2_1.fs
TYPE=local_resource
TARGET=OFFLINE
STATE=OFFLINE

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       54G  9.0G   42G  18% /
/dev/sda1              99M   31M   64M  33% /boot
tmpfs                 2.5G  176M  2.3G   8% /dev/shm

# Mounting Filesystem with OHAS:
$ crsctl start resource acfs.orahome11gR2_1.fs
CRS-2672: Attempting to start 'acfs.orahome11gR2_1.fs' on 'macedonia'
CRS-2676: Start of 'acfs.orahome11gR2_1.fs' on 'macedonia' succeeded

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       54G  9.0G   42G  18% /
/dev/sda1              99M   31M   64M  33% /boot
tmpfs                 2.5G  176M  2.3G   8% /dev/shm
/dev/asm/db112_dbh1-220
                       15G  10M   15G  1% /u01/app/oracle/product/11.2.0/dbhome_1

$ crsctl status resource acfs.orahome11gR2_1.fs
NAME=acfs.orahome11gR2_1.fs
TYPE=local_resource
TARGET=ONLINE
STATE=ONLINE on macedonia

Trying stop DISKGROUP without umount filesystem

$ srvctl stop diskgroup -g DG_ORAHOME11GR2_1
PRCR-1065 : Failed to stop resource ora.DG_ORAHOME11GR2_1.dg
CRS-2529: Unable to act on 'ora.DG_ORAHOME11GR2_1.dg' because that would require stopping or relocating 'acfs.orahome11gR2_1.fs', but the force option was not specified

Now you can restart your server and make sure wich the filesystem will be mounted at startup of OHAS.
If everything is ok you can install Oracle Software on ACFS.

I created a Database (db11g) using ORACLE_HOME on ACFS.

When DBCA create a Database (Oracle Restart) he doesn’t create dependencies between ACFS Mount and Database.

So, when OHAS start a Database (db11g)  he will try start only DISKGROUPs where database files are and just it. As ORACLE_HOME is not mounted Database will not start automatically.

We must create a dependencie between ACFS Mount and Database (like oracle does with Diskgroup).

Using Resource name of Database (ora.db11g.db) let’s create dependencies.

To Start Database (db11g) the ACFS Filesystem must be mounted. And to Stop ACFS Filesystem Database must be OFFLINE. If we try start Database without ACFS Filesystem mounted  OHAS will try start ACFS Filesystem before start Database.

$ crsctl modify resource ora.db11g.db -attr "\
START_DEPENDENCIES='hard(acfs.orahome11gR2_1.fs)pullup(acfs.orahome11gR2_1.fs)',\
STOP_DEPENDENCIES='hard(acfs.orahome11gR2_1.fs)'"

Creating this dependencies does not affect config of Database in OHAS.

$ srvctl config database -d db11g -a
Database unique name: db11g
Database name: db11g
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DG_DATA/db11g/spfiledb11g.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Disk Groups: DG_DATA,DG_FRA
Services:
Database is enabled

Testing:

All Online:

crsctl status resource -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
acfs.orahome11gR2_1.fs
               ONLINE  ONLINE       macedonia
ora.DG_DATA.dg
               ONLINE  ONLINE       macedonia
ora.DG_FRA.dg
               ONLINE  ONLINE       macedonia
ora.DG_ORAHOME11GR2_1.dg
               ONLINE  ONLINE       macedonia
ora.LISTENER.lsnr
               ONLINE  ONLINE       macedonia
ora.asm
               ONLINE  ONLINE       macedonia                Started
ora.ons
               OFFLINE OFFLINE      macedonia
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       macedonia
ora.db11g.db
      1        ONLINE  ONLINE       macedonia                Open
ora.diskmon
      1        ONLINE  ONLINE       macedonia
ora.evmd
      1        ONLINE  ONLINE       macedonia

Stopping Diskgroup DG_ORAHOME11GR2_1

$ srvctl stop database -d db11g
$ crsctl stop resource acfs.orahome11gR2_1.fs
CRS-2673: Attempting to stop 'acfs.orahome11gR2_1.fs' on 'macedonia'
CRS-2677: Stop of 'acfs.orahome11gR2_1.fs' on 'macedonia' succeeded
$ srvctl stop diskgroup -g DG_ORAHOME11GR2_1

### Database, Filesystem and Diskgroup is Offline
$ crsctl status resource -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
acfs.orahome11gR2_1.fs
               OFFLINE OFFLINE      macedonia
ora.DG_DATA.dg
               ONLINE  ONLINE       macedonia
ora.DG_FRA.dg
               ONLINE  ONLINE       macedonia
ora.DG_ORAHOME11GR2_1.dg
               OFFLINE OFFLINE      macedonia
ora.LISTENER.lsnr
               ONLINE  ONLINE       macedonia
ora.asm
               ONLINE  ONLINE       macedonia                Started
ora.ons
               OFFLINE OFFLINE      macedonia
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       macedonia
ora.db11g.db
      1        OFFLINE OFFLINE                               Instance Shutdown
ora.diskmon
      1        ONLINE  ONLINE       macedonia
ora.evmd
      1        ONLINE  ONLINE       macedonia

Starting only database, all resources dependents must be UP automatically.


$ srvctl start database -d db11g

$ crsctl status resource -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
acfs.orahome11gR2_1.fs
               ONLINE  ONLINE       macedonia
ora.DG_DATA.dg
               ONLINE  ONLINE       macedonia
ora.DG_FRA.dg
               ONLINE  ONLINE       macedonia
ora.DG_ORAHOME11GR2_1.dg
               ONLINE  ONLINE       macedonia
ora.LISTENER.lsnr
               ONLINE  ONLINE       macedonia
ora.asm
               ONLINE  ONLINE       macedonia                Started
ora.ons
               OFFLINE OFFLINE      macedonia
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       macedonia
ora.db11g.db
      1        ONLINE  ONLINE       macedonia                Open
ora.diskmon
      1        ONLINE  ONLINE       macedonia
ora.evmd
      1        ONLINE  ONLINE       macedonia

Everthing is fine.

This solution is not supported by Oracle this is a Workaround to start ACFS Filesystem at boot of Server

Enjoy…

Advertisements

8 Comments on “Oracle ACFS Filesystem managed by OHAS on Oracle Restart”

  1. john says:

    Hi,

    This was my exam question, still not sure about the answer.
    Any help is much appreciatedl

    Which three fragments will complete this statement correctly ?
    In a cluster environmet, an acfs volume

    a-)will be automatically mounted by a node on rebooy by default
    b-) must be manually mounted after a node reboot
    c-) will be automatically mounted by a node if it is defined as cluster stack startup if it is included in the ACFS mount registry.
    d-)will be automatically mounted to all node if it is defined as cluster resource when dependent cluster resource requires access
    e-)will be automatically mounted to all node in the cluster when the file system is registered
    f-)must be mounted before it can be registered.

    Like

    • Levi Pereira says:

      Hi,
      The Oracle ACFS mount registry supports both Oracle Restart and Oracle Grid Infrastructure cluster configurations. File systems that are to be mounted persistently (across reboots) can be registered with the Oracle ACFS mount registry. In cluster configurations, registered Oracle ACFS file systems are automatically mounted by the mount registry, similar to a clusterwide mount table. However, in Oracle Restart configurations the automatic mounting of registered Oracle ACFS file systems is not supported.
      By default, an Oracle ACFS file system that is inserted into the cluster mount registry is automatically mounted on all cluster members, including cluster members that are added after the registry addition. However, the cluster mount registry also accommodates single-node and multi-node (subset of cluster nodes) file system registrations. The mount registry actions for each cluster member mount only registered file systems that have been designated for mounting on that member.

      Answer:
      c:)
      d:)
      e:)

      Regards,
      Levi Pereira

      Like

  2. john says:

    Hi Levi,

    Appreciate your help and explanation.
    I know that we can mount acfs volumes using asmca.
    Can we also mount acfs volumes using asmcmd utility ?

    Like

  3. john says:

    Hi Levi,

    Appreciate your time and assistance.
    One last question about adding a node in rac environment

    In rac environment, I know that I can add a node by using “AddNode.sh” script and “clone.pl -silent”
    I am just wondering whether this is also possible with Oracle Enterprise Manager or Oracle Universal Installer using runInstaller -clone ?

    Like

  4. […] (It's not supported, Oracle support fully management of ACFS resources in Oracle Clusterware only) Oracle ACFS Filesystem managed by OHAS on Oracle Restart In your case the script needs to be much more simple than what I created in example on guide […]

    Like

  5. josh says:

    great post, thanks for sharing your work

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s