Troubleshooting: Oracle Database on AIX 6.1
Posted: 08/01/2011 Filed under: AIX, ASM - Automatic Storage Management, Clusterware / Grid Infrastructure, Grid Infrastructure Standalone, My Oracle Support (Metalink), Operating System, Real Application Clusters 2 CommentsRecently installed Oracle Database on AIX 6.1 system I’ll post here the problems and solutions encountered during implementation.
As problems arise I’ll post new problems and solutions.
Installing Oracle Database 11gR2 (11.2.0.3.0) on AIX fail at Prerequisites Check. Missing : IZ97457 and IZ89165
IZ97457: SEA/VIO CLIENT HANG – INTERRUPTS DO NOT REACH THE TRUNK ADAPTER APPLIES TO AIX 6100-05
IZ89165: BIND64 CORES WITH -BLAZY OPTION ON 61TL6 APPLIES TO AIX 7100-00
An Authorized Program Analysis Report, or APAR, is a formal report to IBM development of a problem caused by a suspected defect in a current release of an IBM program.
IBM release an new APAR number for each TL.
IZ97457: SEA/VIO CLIENT HANG - INTERRUPTS DO NOT REACH THE TRUNK ADAPTER APPLIES TO AIX 6100-05 5300-11 - use AIX APAR IZ98424 5300-12 - use AIX APAR IZ98126 6100-04 - use AIX APAR IZ97605 6100-05 - use AIX APAR IZ97457 6100-06 - use AIX APAR IZ96155 7100-00 - use AIX APAR IZ97035 IZ89165: BIND64 CORES WITH -BLAZY OPTION ON 61TL6 APPLIES TO AIX 7100-00 6100-03 - use AIX APAR IZ89304 6100-04 - use AIX APAR IZ89302 6100-05 - use AIX APAR IZ89300 6100-06 - use AIX APAR IZ89514 7100-00 - use AIX APAR IZ89165
In your case Oracle OUI don’t know if APAR is installed because it check only one APAR Number.
So see this example:
root:/> oslevel -r
6100-06
#### Check if apar number IZ89165 is installed.
### Using APAR number required by Oracle
root:/> instfix -a -i |grep IZ89165
#### nothing found
root:/> instfix -a -i |grep IZ89514
IZ89514 Abstract: Linking with -blazy and -G or -brtl causes core dump
IZ89514 Symptom Text:
All filesets for IZ89514 were found.
So, just Ignore warning, if you make sure which APAR already installed.
Install Oracle 10g R2 on AIX 6.1 and the installation hangs at 83% in Prereqs Check.
I have installed the 11.2.0.2 Grid Infrastructure in different User and directory on the same system.
This helps in the administration and security. But I do not recommend doing this because many problems are still solved with workaround.
It was unable to connect due to different OS Users and Groups for the different installation
Linking Fails With “ld: 0706-010 The binder was killed by a signal: Segmentation fault” On AIX 6.1
INFO:
ld: 0706-010 The binder was killed by a signal: Segmentation fault
Check for binder messages or use local problem reporting procedures.
INFO: make: 1254-004 The error code from the last command is 254.
Stop.
INFO: make: 1254-004 The error code from the last command is 2.
Stop.
INFO: End output from spawned process.
INFO: ----------------------------------
INFO: Exception thrown from action: make
Exception Name: MakefileException
Exception String: Error in invoking target 'agent nmo nmb' of makefile '/u01/app/oracle/product/10.2.0/db_1/sysman/lib/ins_sysman.mk'.
Solution:
This a bug on AIX 6.1
IZ88711: BIND64 CORES WITH -BLAZY OPTION ON AIX61
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ88711
Check this note on MOS:
Linking Fails With “ld: 0706-010 The binder was killed by a signal: Segmentation fault” On AIX 6.1 [ID 1264074.1]
If you already in the TL 06, I have the address to download the fix. Only apply the fix.
ftp://public.dhe.ibm.com/aix/efixes/iz88711/
Unable to create 10.2.0.4 database on 11.2.0.2 Grid Infrastructure on AIX 6.1
DBCA displays the following error:
" In order to use Automatic Storage Management (ASM), the ORacle CLuster Synchronization Service (CSS) must be up and running. Run $ORACLE_HOME/localconfig reset script from command prompt to start CSS. The trace file in $ORACLE_HOME/cfgtoollogs/dbca contains the following error: [AWT-EventQueue-0] [15:41:8:479] [StorageOptionsPage.validate:522] CSS Not Running
Solution:
Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment [ID 948456.1]
Due to unpublished bug 8288940, DBCA will fail if database files are located in ASM.
As of now, patch 8288940 is available for 10.2.0.4, 11.1.0.6 and 11.1.0.7 as a .jar file as it is platform independent; fix will be included in the 10.2.0.5 patchset.
OUI FAILS TO DETECT RUNNING PROCESSES CORRECTLY ON AIX 6.1 ML 4 SP1
Oracle Universal Installer has detected that there are processes running in the currently selected Oracle Home. The following processes need to be shutdown before continuing:
java
This a bug on AIX.
IZ67400: FUSER GIVES INCORRECT PIDS WITH -X OPTION
Solution:
Check this note on MOS:
The Installer (OUI) Detects Processes Running on ORACLE_HOME on AIX 6.1 TL04 SP1 [ID 980602.1]
Or
http://forums.oracle.com/forums/thread.jspa?messageID=4070170�
Installing the Oracle Database 10gR2 (10.2.0.1.0) on AIX 6.1 error OUI-18001 is raised
OUI-18001: The operating system 'AIX Version 6100.01' is not supported.
Check all pre-requesistes in documentation.
http://download.oracle.com/docs/cd/B19306_01/install.102/b19075/pre_install.htm
And install 10gR2 (10.2.0.1.0), invoking the OUI (runInstaller) using “-ignoreSysPrereqs” option.
./runInstaller -ignoreSysPrereqs
Levi Pereira | Oracle Blog 
We got our first IBM Power 7 Servers back in April 2010 with AIX 6.1 TL 05 installed. The pSeries were configured with VIOS to safe costs.
We had 2 LPARs on 2 Power 7 over which we created a RAC for our mission critical ERP system.
I can not remember how many nights we passed finding the root cause why the Ethernet adapter suddenly went down. All vendors were fingerpointing each other (Oracle, Cisco, EMC & IBM).
We even involved IBM/Oracle joint solution centre guys from Montpellier.
As out of our experience from the past with IBM VIOS being buggy and unreliable, requiring alot of patches, we searched IBM KB 1 day and found IZ97457. So after we applied this instfix everything worked.
This is not really what you wanna experience when you buy a 400’000 EUR box.
Some things obviously never change @ IBM. Since I first ran Oracle on Power, IBM has been a patching bitch when it comes to Oracle (worse than Mircosoft). You should take therefore budget alot of time for proactive monitoring and patching. Therfore when evaluating IBM Pseries also calculate a fair amount of this time into TCO.
IBM Virtualization for IBM Power (similiar to System Z) has great capbilities and is a powerful API. However mainly this can cause issues, if sysadmins and dbas do not talk to each other making each other understand what the limits are of each technology.
The following golden rules for running Oracle Database on IBM Pseries with Virtualization are based should help most people to avoid alot of trouble:
1. A Database is a static thing, therefore it is only limited aware of Virtualization. It bases it’s information on it’s statistics, so if your sysadmin plays with the entitled capacity values or you run in uncapped mode, this can lead to weird execution plans and/or situation, where db picks a wrong execution plan, because it thinks “hey I’ve got 16 CPU cores”, but in reality it only has 4 VCPU on a IBM Power 7, which has 4 threads and an entitled capacity of 0.5, but Oracle’s parameter file value shows cpu_count=16, as oracle “sees” 4 VCPU * 4 Threads = 16 CPU. Oracle does not know this server is virtual.
2. IBM VIOS is a buggy thing that requires alot of testing and many patches on VIOS and Client LPARS as indicated above. In many cases you will have more troubles than benefits when using VIOS with Oracle / SAP deployments. So at some point you need to decide, wheither you want to use Oracle technologies (ASM, RAC, Data Guard) or IBM Technologies (HACMP, Live Partition Mobility) for Clustering and HA. I personally do not recommend to use VIOS with Oracle RAC / ASM in production. Generally carefully evaluate the pros and cons of using VIOS for Oracle / SAP deployments.
3. Even tough we have alot possibilities, keep it simple and easy to manage. Don’t jump right away of new technologies. Remember: the database is the core of all business applications. Nobody will complain, if a web server is offline for 30 secs, but if your db goes down, everybody will notice it, therefore make sure you have done your homework.
When using Oracle Technologies, the following setup is a good one to start and works for IBM p5 and above:
a) OLTP systems (10g or higher)
SYSTEM: No VIOS
LPAR: Dedicated Processor or Capped entitlement, no memory sharing
HA: Oracle Data Guard physical standby, as it protects against server, storage and site failures
FS: Oracle ASM, as we can add, remove and balance I/O while db’s are running (no need to stop db like with ext3 / JFS to do certain resize operations due still active oracle processes)
Memory can be added on the Fly, but not removed.
To increase entitlement / VCPU, shut down db on affected LPAR, increase entitlement / VCPU, start db.
b) Data Warehouses (10g or higher)
SYSTEM: No VIOS
LPAR: Dedicated Processor or Capped entitlement, no memory sharing
HA: Depending on the requirements and your available SAN infrastructure a good backup and a empty LPAR to which perform a daily duplicate / snapshot copy may be suffiient. You could even Oracle Data Guard as well DW.
With Oracle 11gR2 you have the following addtional possibilities:
– Use Active Standby Feature for Dataguard to offload reporting to standby machine
– Use RAC one Node for a Active-Passive Cluster (cheaper than full RAC licenses)
FS: Oracle ASM, as we can add, remove and balance I/O while db’s are running (no need to stop db like with ext3 / JFS to do certain resize operations due still active oracle processes)
Memory can be added on the Fly, but not removed.
To increase entitlement / VCPU, shut down db on affected LPAR, increase entitlement / VCPU, start db.
LikeLike
Thank you Efstathios.
LikeLike