Using ACFS Snapshot to create a consistent recovery point of ORACLE_HOME

ACFS Snapshots

ACFS provides snapshotting capability for the respective filesystem. This snapshot uses the First Copy-on-Write (FCOW) methodology to enable a consistent, version-based, online view of the source filesystem.
Snapshots are immutable views of the source file system as it appeared at a specific point in time.
Snapshots are initially a sparse filesystem, as the source filesystem files change, the before-image extent of that file is copied into the snapshot directory. The before-image granularity is an ACFS extent, thus if any byte in an extent is modified, the extent is COW’ed and any subsequent changes in that extent requires no action for the snap.

I’ll do some tests using the ACFS Snapshot.

I go apply the Critical Patch Update July 2011 (ID 12419321) on ORACLE_HOME (11.2.0.2) and apply CPU on Database (SQL files).

After finish Patch ORACLE_HOME and Database I go perform recovery point in ORACLE_HOME and Database to bring back point before applying a database patchset.

Things I did not like in the ACFS Snapshot.

  • To recover a file we must perform using standard file copy or replace commands from O.S . (No exists restore command to ACFS Snapshot)
  • I cannot change default location of Snapshots,  if no space available to Snapshot the  filesystem will be exhausted.
Let’s Start.

Backup a Database

I’ll not use Restore Database using (Restore Database ) from RMAN , I will use feature GRP  (guaranteed restore point) is much more fast than Restore all Database files.

It is mandatory do Backup of Database through RMAN, even using GRP because GRP must be discarded after upgrade/downgrade.

$ export NLS_DATE_FORMAT='Mon DD YYYY HH24:MI:SS'
$ rman target /
Recovery Manager: Release 11.2.0.2.0 - Production on Fri Jul 29 11:49:29 2011
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
connected to target database: DB11G (DBID=227286804)

RMAN> shutdown immediate;
using target database control file instead of recovery catalog
database closed
database dismounted
Oracle instance shut down

RMAN> startup mount
connected to target database (not started)
Oracle instance started
database mounted

Total System Global Area    1185853440 bytes
Fixed Size                     2225912 bytes
Variable Size                788531464 bytes
Database Buffers             385875968 bytes
Redo Buffers                   9220096 bytes

RMAN> backup database plus archivelog delete input;

Starting backup at Jul 29 2011 11:50:13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=194 device type=DISK
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=12 RECID=1 STAMP=757770541
...
Finished backup at Jul 29 2011 11:50:18

Starting backup at Jul 29 2011 11:50:18
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=+DG_DATA/db11g/datafile/system.256.757617145
...
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:25
...
Finished backup at Jul 29 2011 11:50:47

RMAN> exit

After backup finish leave Database in Mount State.

Guaranteed Restore Points

Like a normal restore point, a guaranteed restore point serves as an alias for an SCN in recovery operations.
A principal difference is that guaranteed restore points never age out of the control file and must be explicitly dropped. In general, you can use a guaranteed restore point as an alias for an SCN with any command that works with a normal restore point. Except as noted, the information about where and how to use normal restore points applies to guaranteed restore points as well.

A guaranteed restore point ensures that you can use Flashback Database to rewind a database to its state at the restore point SCN, even if the generation of flashback logs is disabled. If flashback logging is enabled, then a guaranteed restore point enforces the retention of flashback logs required for Flashback Database to any SCN after the earliest guaranteed restore point. Thus, if flashback logging is enabled, you can rewind the database to any SCN in the continuum rather than to a single SCN only.

http://download.oracle.com/docs/cd/E11882_01/backup.112/e10642/flashdb.htm#CFHGFGFH

Let’s use feature Guaranteed Restore Points to save time at moment of restore.

$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.2.0 Production on Fri Jul 29 14:56:45 2011
Copyright (c) 1982, 2010, Oracle.  All rights reserved.

SQL> CREATE RESTORE POINT before_upgrade_12419321 GUARANTEE FLASHBACK DATABASE;
Restore point created.

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit

Creating Snapshot of ORACLE_HOME

The snapshot must be created with root privileges.

$ sudo /sbin/acfsutil snap create pre_patch_12419321 /u01/app/oracle/product/11.2.0/dbhome_1
acfsutil snap create: Snapshot operation is complete.

$ sudo /sbin/acfsutil info fs /u01/app/oracle/product/11.2.0/dbhome_1
/u01/app/oracle/product/11.2.0/dbhome_1
    ACFS Version: 11.2.0.2.0
    flags:        MountPoint,Available
    mount time:   Thu Jul 28 15:45:27 2011
    volumes:      1
    total size:   16106127360
    total free:   11288350720
    primary volume: /dev/asm/db112_dbh1-220
        label:
        flags:                 Primary,Available,ADVM
        on-disk version:       39.0
        allocation unit:       4096
        major, minor:          252, 112641
        size:                  16106127360
        free:                  11288350720
        ADVM diskgroup         DG_ORAHOME11GR2_1
        ADVM resize increment: 268435456
        ADVM redundancy:       unprotected
        ADVM stripe columns:   4
        ADVM stripe width:     131072
    number of snapshots:  1
    snapshot space usage: 19959808

The Database and ORACLE_HOME are ready for patching.

Patching ORACLE_HOME

$ opatch lsinventory
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /u01/app/oracle/product/11.2.0/dbhome_1/oui
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2011-07-29_12-08-55PM.log

Patch history file: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt
Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2011-07-29_12-08-55PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.2.0
There are 1 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.
--------------------------------------------------------------------------------
OPatch succeeded.

$ ls -l
total 3142372
drwxrwxr-x 17 oracle oinstall       4096 May 26 10:20 12419321
-rw-r--r--  1 oracle oinstall    2982182 Jul 29 11:35 p12419321_112020_Linux-x86-64.zip
$ cd 12419321/

$ opatch napply -skip_subset -skip_duplicate
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /u01/app/oracle/product/11.2.0/dbhome_1/oui
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2011-07-29_12-10-03PM.log

Patch history file: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt

Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking skip_duplicate
Checking skip_subset
Checking conflicts against Oracle Home...
OPatch continues with these patches:   11830776  11830777  11830778  12419321  12586486  12586487  12586488  12586489  12586490  12586491  12586492  12586493  12586494  12586495  12586496

Do you want to proceed? [y|n]
y
User Responded with: Y

Running prerequisite checks...

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/11.2.0/dbhome_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
.....
Verifying the update...
Inventory check OK: Patch ID 12586496 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 12586496 are present in Oracle Home.
Running make for target client_sharedlib
Running make for target client_sharedlib
Running make for target ioracle

The local system has been patched and can be restarted.

UtilSession: N-Apply done.

OPatch succeeded.

$ opatch lsinventory
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /u01/app/oracle/product/11.2.0/dbhome_1/oui
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2011-07-29_14-04-23PM.log

Patch history file: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2011-07-29_14-04-23PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.2.0
There are 1 products installed in this Oracle Home.

Interim patches (15) :

Patch  12586496     : applied on Fri Jul 29 12:16:24 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 06:02:42 hrs PST8PDT
   Bugs fixed:
     12586496

Patch  12586495     : applied on Fri Jul 29 12:16:17 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 06:00:31 hrs PST8PDT
   Bugs fixed:
     12586495

Patch  12586494     : applied on Fri Jul 29 12:16:11 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 05:58:48 hrs PST8PDT
   Bugs fixed:
     12586494

Patch  12586493     : applied on Fri Jul 29 12:15:58 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 05:55:56 hrs PST8PDT
   Bugs fixed:
     12586493

Patch  12586492     : applied on Fri Jul 29 12:15:55 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 05:52:39 hrs PST8PDT
   Bugs fixed:
     12586492

Patch  12586491     : applied on Fri Jul 29 12:15:42 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 05:46:21 hrs PST8PDT
   Bugs fixed:
     12586491

Patch  12586490     : applied on Fri Jul 29 12:15:35 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 05:35:20 hrs PST8PDT
   Bugs fixed:
     12586490

Patch  12586489     : applied on Fri Jul 29 12:15:08 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 05:29:23 hrs PST8PDT
   Bugs fixed:
     12586489

Patch  12586488     : applied on Fri Jul 29 12:15:04 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 05:27:10 hrs PST8PDT
   Bugs fixed:
     12586488

Patch  12586487     : applied on Fri Jul 29 12:14:51 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 05:24:30 hrs PST8PDT
   Bugs fixed:
     12586487

Patch  12586486     : applied on Fri Jul 29 12:14:27 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 03:35:49 hrs PST8PDT
   Bugs fixed:
     12586486

Patch  12419321     : applied on Fri Jul 29 12:14:03 BRT 2011
Unique Patch ID:  13783102
   Created on 26 May 2011, 06:16:06 hrs PST8PDT
   Bugs fixed:
     12419321, 11724984

Patch  11830778     : applied on Fri Jul 29 12:14:01 BRT 2011
Unique Patch ID:  13783102
   Created on 4 Apr 2011, 02:41:43 hrs PST8PDT
   Bugs fixed:
     11830778

Patch  11830777     : applied on Fri Jul 29 12:13:46 BRT 2011
Unique Patch ID:  13783102
   Created on 4 Apr 2011, 02:36:22 hrs PST8PDT
   Bugs fixed:
     11830777

Patch  11830776     : applied on Fri Jul 29 12:13:17 BRT 2011
Unique Patch ID:  13783102
   Created on 4 Apr 2011, 02:27:33 hrs PST8PDT
   Bugs fixed:
     11830776

--------------------------------------------------------------------------------
OPatch succeeded.

Patching Database


$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.2.0 Production on Fri Jul 29 12:19:12 2011
Copyright (c) 1982, 2010, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1185853440 bytes
Fixed Size                  2225912 bytes
Variable Size             788531464 bytes
Database Buffers          385875968 bytes
Redo Buffers                9220096 bytes
Database mounted.
Database opened.
SQL> @$ORACLE_HOME/rdbms/admin/catbundle.sql cpu apply

PL/SQL procedure successfully completed.

Executing script file...

.....

Session altered.

SQL> COMMIT;

Commit complete.

SQL> SPOOL off
SQL> SET echo off
Check the following log file for errors:
/u01/app/oracle/cfgtoollogs/catbundle/catbundle_CPU_DB11G_APPLY_2011Jul29_12_20_14.log
SQL> exit

SQL> select * from registry$history where BUNDLE_SERIES='CPU';

ACTION_TIME                    ACTION     NAMESPACE  VERSION     ID COMMENTS                       BUNDLE_SERIES
------------------------------ ---------- ---------- ---------- --- ------------------------------ ------------------------------
29-JUL-11 03.08.31.254298 PM   APPLY      SERVER     11.2.0.2     2 CPUJul2011                     CPU

Performing Patch Deinstallation

Peforming restore point of Database using GRP.

I love the feature GRP, I have performed the restore in 3 seconds.
Using command RESTORE of RMAN depending on the size of the database could take hours

SELECT NAME, SCN, TIME, DATABASE_INCARNATION# DI,GUARANTEE_FLASHBACK_DATABASE, STORAGE_SIZE
FROM V$RESTORE_POINT
NAME                                     SCN TIME                                        DI GUA STORAGE_SIZE
------------------------------ ------------- ----------------------------------- ---------- --- ------------
BEFORE_UPGRADE_12419321              1236543 29-JUL-11 02.57.54.000000000 PM              3 YES     15941632

RMAN> shutdown immediate;
using target database control file instead of recovery catalog
database closed
database dismounted
Oracle instance shut down

RMAN> startup mount

connected to target database (not started)
Oracle instance started
database mounted

Total System Global Area    1185853440 bytes

Fixed Size                     2225912 bytes
Variable Size                788531464 bytes
Database Buffers             385875968 bytes
Redo Buffers                   9220096 bytes

RMAN> FLASHBACK DATABASE TO RESTORE POINT BEFORE_UPGRADE_12419321;

Starting flashback at Jul 29 2011 15:22:52
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=194 device type=DISK

starting media recovery
media recovery complete, elapsed time: 00:00:03

Finished flashback at Jul 29 2011 15:22:56

RMAN> alter database open resetlogs;
database opened

SQL> select * from registry$history;
ACTION_TIME                         ACTION     NAMESPACE  VERSION            ID COMMENTS             BUNDLE_SERIES
----------------------------------- ---------- ---------- ---------- ---------- -------------------- ---------------
05-SEP-10 06.22.14.370943 AM        APPLY      SERVER     11.2.0.2            0 Patchset 11.2.0.2.0  PSU
27-JUL-11 05.14.47.164924 PM        APPLY      SERVER     11.2.0.2            0 Patchset 11.2.0.2.0  PSU

SQL> shutdown immmediate;

Recover the filesystem (ORACLE_HOME) leveraging the snapshot

I use the cp command from O.S and you must use flag -p (to preserve mode,ownership,timestamps) and flag -R (to copy directories recursively) using root privileges.

I took 16 minutes to copy all files from Snapshot to ORACLE_HOME. (the copy took a long time)

$ sudo time cp -Rp /u01/app/oracle/product/11.2.0/dbhome_1/.ACFS/snaps/pre_patch_12419321/* /u01/app/oracle/product/11.2.0/dbhome_1/

$ opatch lsinventory
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /u01/app/oracle/product/11.2.0/dbhome_1/oui
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2011-07-29_15-45-53PM.log

Patch history file: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2011-07-29_15-45-53PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.2.0
There are 1 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

--------------------------------------------------------------------------------
OPatch succeeded.

Restoring with the conventional method using OPatch

The OPatch took 5 minutes to rollback the patch.


$ opatch nrollback -idFile $ORACLE_HOME/cpu/CPUJul2011/rollback_all.lst
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /u01/app/oracle/product/11.2.0/dbhome_1/oui
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2011-07-29_15-54-23PM.log

Patch history file: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt

Invoking utility "nrollback"

Patches will be rolled back in the following order:
   11830776   11830777   11830778   12419321   12586486   12586487   12586488   12586489   12586490   12586491   12586492   12586493   12586494   12586495   12586496

Running prerequisite checks...
The following patch(es) will be rolled back: 11830776  11830777  11830778  12419321  12586486  12586487  12586488  12586489  12586490  12586491  12586492  12586493  12586494  12586495  12586496

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/11.2.0/dbhome_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NRollback' for restore. This might take a while...

Rolling back patch 11830776...

RollbackSession rolling back interim patch '11830776' from OH '/u01/app/oracle/product/11.2.0/dbhome_1'

....

Running make for target client_sharedlib
Running make for target client_sharedlib
Running make for target ioracle

The local system has been patched and can be restarted.

UtilSession: N-Rollback done.

OPatch succeeded.

$ opatch lsinventory
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /u01/app/oracle/product/11.2.0/dbhome_1/oui
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2011-07-29_16-01-27PM.log

Patch history file: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2011-07-29_16-01-27PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.2.0
There are 1 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

--------------------------------------------------------------------------------
OPatch succeeded.

Clean Up

After the maintenance is mandatory to clean, because if we not disable ACFS SNAPSHOT and GRP, it will continue to generate data for each modification of the database and files in Oracle home.

$ rman target /
Recovery Manager: Release 11.2.0.2.0 - Production on Tue Aug 9 14:55:14 2011
connected to target database: DB11G (DBID=227286804)

RMAN>  LIST RESTORE POINT ALL;
using target database control file instead of recovery catalog
SCN              RSP Time  Type       Time      Name
---------------- --------- ---------- --------- ----
1236543                    GUARANTEED 29-JUL-11 BEFORE_UPGRADE_12419321

$sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Tue Aug 9 14:57:18 2011
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production

SQL> DROP RESTORE POINT BEFORE_UPGRADE_12419321;
Restore point dropped.

$sudo /sbin/acfsutil snap delete  pre_patch_12419321 /u01/app/oracle/product/11.2.0/dbhome_1
acfsutil snap delete: Snapshot operation is complete.

Conclusion

The snapshot feature is good when updated/patch software or files that do not allow the rollback  on case of failures.

I still prefer the conventional method using OPATCH to perform the rolling back a patch.

Rolling back a patch using the OPatch utility is more safe and faster than copying files fromACFS Snapshot. OPatch performs all tasks necessary to leave the ORACLE_HOME with integrity. The copy of data via the CP command must be executed with caution and we should run the relink the binaries (if necessary) manually.

When we applying patch the utility OPATCH change few binary files on Oracle Home, but Oracle files are relinked by OPATCH, so at this moment a lot files are changed and all this files changed will be copied into the snapshot directory with before-image extent.

So, when we rolling back all files using ACFS SNAPSHOT a lot files will be copied to Oracle Home, because that  OPATCH to rolling back it’s more fast, he back few files and relink again all files on Oracle Home.

Enjoy..

Advertisements


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s