How configure Multiples Public Network in a Grid Infrastructure 11g R2 (11.2) environment

Introduction

Using earlier versions of RAC Oracle 11.2 is not possible to create more than one public subnet to serve different database clients in different sites/network, to get the routing of connections we should have help from a software/hardware from third parties.

Now it’s possible configure multiples network segment (i.e different subnet) with GI/RAC 11.2.
Oracle GI/RAC 11.2 works like a router managing connections of multiple network, routing connections to the correct network/endpoint.

During the Grid Infrastructure installation, it’s not possible configure more than one public/vip network. So, the other networks should be configured manually after installation.

Purpose

The purpose of this post is to explain how to create  multipe network/listener/services on multiple public network in an 11.2 Grid Infrastructure environment.

No downtime is required to perform this task. All tasks can be done with Clusterware and RAC online.

Concept before Starting

Please Note – Limitations of Grid Infrastructure 11g R2:

SCAN feature will work only in one public network, because we can configure only one SCAN for entire Clusterware Stack and SCAN must be associated whith only one public network. So, only one  public network  we can use all feature  of GI 11.2 the others public networks will be required use the VIP in our Oracle Clients  like in version 11g R1 and 10g.

I believe that Oracle will solve this issue in later versions.

Before start configuration you must learn how things work to know to properly configure our RAC.  You need to know how they work to understand how to configure your Oracle RAC.


Basic Concept:

The listener is a server-side process that listens for incoming client connection requests and manages traffic to the database. When a database instance starts, and at various times during its life, the instance contacts a listener and establishes a communication pathway to this instance.

Service registration enables the listener to determine whether a database service and its service handlers are available. A service handler is a dedicated server process or dispatcher that acts as a connection point to a database. During registration, the PMON process provides the listener with the instance name, database service names, and the type and addresses of service handlers. This information enables the listener to start a service handler when a client request arrives.

A service name is a logical representation of a service used for client connections.

When a client connects to a listener, it requests a connection to a service.Thus, the listener acts as a mediator between the client and instances and routes the connection request to the right place.

To ensure service registration works properly,  the initialization parameter file should contain the following parameters:

  • LOCAL_LISTENER for the local listener
  • REMOTE_LISTENER for the remote listener

LOCAL_LISTENER: To have PMON register with a local listener in RAC env we must configure the LOCAL_LISTENER parameter in the initialization parameter file to locate the local listener. Multiple addresses are supported, but connect-time failover and client load balancing features are not supported.

REMOTE_LISTENER: A remote listener is a listener residing on one computer that redirects connections to a database instance on another computer. In a dedicated server RAC environment, we must enable the PMON background process to register with a remote listener. You do this by configuring the REMOTE_LISTENER parameter.

Each network must have a dedicated LISTENER.  It would be necessary to configure multiple addresses (Listener) in the parameter  LOCAL_LISTENER from all instances  to serve NET1 and NET2. Multiple addresses are supported, but connect-time failover and client load balancing features are not supported. (first problem) 😦

If we use  only parameter LOCAL_LISTENER and REMOTE_LISTENER the connections can be redirected by REMOTE_LISTENER to LOCAL_LISTENER  from others network. (second problem) 😦 😦

To ensure that connections to the remote listener are only redirected to the local listener on the same network Oracle create new Parameter LISTENER_NETWORKS :-), this parameter is avaliable only in version 11g R2.

  • LISTENER_NETWORKS specifies one or more sets of local & remote listeners for cross-registration. All listeners within the same network_name will cross-register.

Using parameter LISTENER_NETWORKS we can configure multipes LOCAL & REMOTE Listener in your own network.  It ensure wich connections incoming by network net1 will be redirected over RAC Database only on network net1.

This tasks will be done in 2 steps:

1. Create New Network and Listener in Oracle Clusterware

2. Configure RAC to support multipe Network and Create Service

Creating  New Network and Listener in Oracle Clusterware

The Clusterware must be configured and online in all nodes.

Overview of Environment

  1. First Public network is called North.
  2. Second Public network is called South.
  3. Third Network is called East.
Infrastructure:

Servers Hostname
1° northora01
2° northora02
3° northora03

North - eth0 - 192.168.217.0
# Public Hostnames/IP North
northora01 - 192.168.217.60
northora02 - 192.168.217.45
northora03 - 192.168.217.68

# Virtual Hostnames/IP North
northora01-vip - 192.168.217.52
northora02-vip - 192.168.217.53
northora02-vip - 192.168.217.69

South - eth2 - 140.120.120.0
# Public Hostnames/IP South
southora01 - 140.120.120.200
southora02 - 140.120.120.201
southora03 - 140.120.120.202
# Virtual Hostnames/IP South
southora01-vip - 140.120.120.100
southora02-vip - 140.120.120.101
southora03-vip - 140.120.120.102

East - eth3 - 154.120.120.0
# Public Hostnames/IP East
eastora01 - 154.120.120.200
eastora02 - 154.120.120.201
eastora03 - 154.120.120.202
# Virtual Hostnames/IP East
eastora01-vip - 154.120.120.100
eastora02-vip - 154.120.120.101
eastora03-vip - 154.120.120.102

Before start the New Interfaces (eth2 and eth3) must be configured with IP,Mask and Gateway.

After setting up the new interfaces you must set VIP hostnames from new networks in hosts files (e.g /etc/hosts) from all nodes and add VIP-hostnames in your DNS.

# /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
# Public North
192.168.217.60          northora01.north.com		northora01
192.168.217.45          northora02.north.com		northora02
192.168.217.68          northora03.north.com		northora03

# VIP North
192.168.217.52          northora01-vip.north.com		northora01-vip
192.168.217.53          northora02-vip.north.com		northora02-vip
192.168.217.69          northora03-vip.north.com		northora03-vip

# Public South
140.120.120.200         southora01.south.com            southora01
140.120.120.201         southora02.south.com            southora02
140.120.120.202         southora03.south.com            southora03

# VIP South
140.120.120.100         southora01-vip.south.com	southora01-vip
140.120.120.101         southora02-vip.south.com	southora02-vip
140.120.120.102        	southora03-vip.south.com	southora03-vip

# Public East
154.120.120.200		eastora01.east.com		eastora01
154.120.120.201		eastora02.east.com		eastora02
154.120.120.202		eastora03.east.com		eastora03

# Vip East
154.120.120.100		eastora01-vip.east.com		eastora01-vip
154.120.120.101		eastora02-vip.east.com		eastora02-vip
154.120.120.102		eastora03-vip.east.com		eastora03-vip


Ensure Public and VIP address is defined for the 2nd and 3nd public network

Creating the CRS resource (network/vip) using srvctl

A new network and new vip resources for the new network are created by using the ‘-k’ switch.
The-k option indicates to which network it is. The public network has automatically installed the first number.

Using Oracle Grid Infrastucture 11.2.0.1 you can’t create network resource explicitly, when we create a new vip resource the network resource will be created implicitly and a dependency between resources (vip and network) will be configured.

Retrieving information about the environment

# Actual Network
# Network Info
crsctl status resource -f |grep NAME=ora.net -A 3
NAME=ora.net1.network
TYPE=ora.network.type
STATE=ONLINE
TARGET=ONLINE

# VIP Info
crsctl status resource -f |grep NAME=ora.northora -A 3 |grep .vip -A 3
NAME=ora.northora01.vip
TYPE=ora.cluster_vip_net1.type
STATE=ONLINE
TARGET=ONLINE
--
NAME=ora.northora02.vip
TYPE=ora.cluster_vip_net1.type
STATE=ONLINE
TARGET=ONLINE
--
NAME=ora.northora03.vip
TYPE=ora.cluster_vip_net1.type
STATE=ONLINE
TARGET=ONLINE

# Getting Nodes Information
olsnodes -n -s
northora01	1       Active
northora02  	2       Active
northora03   	3       Active

Adding Network/VIP Resources using Clusterware 11.2.0.1

as root user:

# srvctl add vip -n  -k  -A //[if1[|if2...]] [-v]

srvctl add vip -n northora01 -A southora01-vip/255.255.255.0/eth2 -k 2
srvctl add vip -n northora02 -A southora02-vip/255.255.255.0/eth2 -k 2
srvctl add vip -n northora03 -A southora03-vip/255.255.255.0/eth2 -k 2

srvctl start vip -i southora01-vip
srvctl start vip -i southora02-vip
srvctl start vip -i southora03-vip

# Checking Status Network 2
crsctl status resource -f |grep NAME=ora.net2 -A 3
NAME=ora.net2.network
TYPE=ora.network.type
STATE=ONLINE
TARGET=ONLINE

crsctl status resource -f |grep NAME=ora.south -A 3 |grep .vip -A 3
NAME=ora.southora01-vip.vip
TYPE=ora.cluster_vip_net2.type
STATE=ONLINE
TARGET=ONLINE
--
NAME=ora.ora.southora02-vip.vip.vip
TYPE=ora.cluster_vip_net2.type
STATE=ONLINE
TARGET=ONLINE
--
NAME=ora.ora.southora03-vip.vip.vip
TYPE=ora.cluster_vip_net2.type
STATE=ONLINE
TARGET=ONLINE

From 11.2.0.2+, network resource can be created explicitly:

as root user:
# srvctl add network [-k ] -S //[if1[|if2...]] [-w ] [-v]

srvctl add network -k 3 -S 154.120.120.0/255.255.255.0/eth3

crsctl status resource -f |grep NAME=ora.net3 -A 3
NAME=ora.net3.network
TYPE=ora.network.type
STATE=OFFLINE
TARGET=OFFLINE

Then add vip resource for the 3nd network:
srvctl add vip -n northora01 -A eastora01-vip/255.255.255.0/eth3 -k 3
srvctl add vip -n northora02 -A eastora02-vip/255.255.255.0/eth3 -k 3
srvctl add vip -n northora03 -A eastora03-vip/255.255.255.0/eth3 -k 3

And starting vip resource
srvctl start vip -i eastora01-vip
srvctl start vip -i eastora02-vip
srvctl start vip -i eastora03-vip

 crsctl status resource -f |grep NAME=ora.east -A 3 |grep .vip -A 3
NAME=ora.eastora01-vip.vip
TYPE=ora.cluster_vip_net3.type
STATE=ONLINE
TARGET=ONLINE
--
NAME=ora.eastora02-vip.vip
TYPE=ora.cluster_vip_net3.type
STATE=ONLINE
TARGET=ONLINE
--
NAME=ora.eastora03-vip.vip
TYPE=ora.cluster_vip_net3.type
STATE=ONLINE
TARGET=ONLINE

Creating New Listener  for each network.

You can create new listener using srvctl, but I recommend create Listener using NETCA.
As the grid user invoke “netca” from the 11.2 GRID_HOME, the select “Cluster configuration” -> “Listener configuration” -> “Add”, enter Listener name as required.

Repeat the procedure above to create the Listener on the network South.

Validating Clusterware Configuration

Checking Listeners created

srvctl config listener -a
Name: LISTENER
Network: 1, Owner: oracle
Home:
  /u01/app/11.2.0/grid on node(s) northora01,northora02,northora03
End points: TCP:1521
Name: LISTENER_SOUTH
Network: 2, Owner: oracle
Home:
  /u01/app/11.2.0/grid on node(s) northora01,northora02,northora03
End points: TCP:1522
Name: LISTENER_EAST
Network: 3, Owner: oracle
Home:
  /u01/app/11.2.0/grid on node(s) northora01,northora02,northora03
End points: TCP:1523

srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): northora01,northora02,northora03
Listener LISTENER_EAST is enabled
Listener LISTENER_EAST is running on node(s): northora01,northora02,northora03
Listener LISTENER_SOUTH is enabled
Listener LISTENER_SOUTH is running on node(s): northora01,northora02,northora03

srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node northora01
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node northora02
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node northora03

srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

The Configuration of  network and listener on Cluster-Wide is finished.

Configuring RAC Database “db11g”  to support multipe Network and Creating Service

Now we need configure Oracle RAC Database (11.2) to provide service on all networks (network 1,2 and 3).

 Prepare the database instance for the new listener

To ensure that connections to the remote listener are only redirected to the local listener on the same network, LISTENER_NETWORKS parameter needs to be set in the pfile or spfile for the database instance.

Use tnsnames.ora from Oracle Home (RAC)  to resolve listener name alias for LISTENER_NETWORKS.  All alias must be in tnsnames.ora from all nodes of RAC Database.

# tnsnames.ora Network Configuration File: /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.
LISTENER_NORTH1=
 (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = northora01-vip.oracle.com)(PORT = 1521))
  )

LISTENER_NORTH2=
 (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = northora02-vip.oracle.com)(PORT = 1521))
  )

LISTENER_NORTH3=
 (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = northora03-vip.oracle.com)(PORT = 1521))
  )

LISTENER_SOUTH1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = southora01-vip.oracle.com)(PORT = 1522))
  )

LISTENER_SOUTH2 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = southora02-vip.oracle.com)(PORT = 1522))
  )

LISTENER_SOUTH3 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = southora03-vip.oracle.com)(PORT = 1522))
  )

REMOTE_SOUTH =
   (DESCRIPTION_LIST =
     (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = southora01-vip.oracle.com)(PORT = 1522)))
     (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = southora02-vip.oracle.com)(PORT = 1522)))
     (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = southora03-vip.oracle.com)(PORT = 1522)))
   )

LISTENER_EAST1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = eastora01-vip.east.com)(PORT = 1523))
  )

LISTENER_EAST2 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = eastora02-vip.east.com)(PORT = 1523))
  )

LISTENER_EAST3 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = eastora03-vip.east.com)(PORT = 1523))
  )

REMOTE_EAST =
   (DESCRIPTION_LIST =
     (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora01-vip.east.com)(PORT = 1523)))
     (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora02-vip.east.com)(PORT = 1523)))
     (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora03-vip.east.com)(PORT = 1523)))
   )

Don’t forget configure sqlnet.ora.

# sqlnet.ora Network Configuration File: /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.ora
# Generated by Oracle configuration tools.

NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)

ADR_BASE = /u01/app/oracle

Testing connectivity of all entries in TNSNAMES.ora using utility tnsping

This step is very important, all of listener alias name must be validated in all nodes.

e.g

tnsping LISTENER_SOUTH1

TNS Ping Utility for Linux: Version 11.2.0.2.0 - Production on 18-OCT-2011 17:31:12

Copyright (c) 1997, 2010, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.ora

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = southora1-vip.oracle.com)(PORT = 1522)))
OK (0 msec)

# Testing REMOTE

tnsping REMOTE_EAST

TNS Ping Utility for Linux: Version 11.2.0.2.0 - Production on 18-OCT-2011 17:31:55

Copyright (c) 1997, 2010, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.ora

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora01-vip.east.com)(PORT = 1523))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora02-vip.east.com)(PORT = 1523))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = eastora03-vip.east.com)(PORT = 1523))))
OK (0 msec)

Configuring Oracle RAC to support multiples network

Please, don’t change  current parameter of LOCAL_LISTENER and REMOTE_LISTENER of your spfile/pfile.

Setting  LISTENER_NETWORKS is enough to RAC to work properly.

Recommendation : Set parameter using scope=both, it’s will validade if the all Listener alias is working. If you use scope=spfile, it will not validaded and if not was configured properly the error will be raised at moment of startup of database instance.

####### Instance 1 #############
ALTER SYSTEM
 SET LISTENER_NETWORKS = '((NAME=NORTH_NET)(LOCAL_LISTENER=LISTENER_NORTH1)(REMOTE_LISTENER=global-scan.north.com:1521))',
                         '((NAME=SOUTH_NET)(LOCAL_LISTENER=LISTENER_SOUTH1)(REMOTE_LISTENER=REMOTE_SOUTH))',
                         '((NAME=EAST_NET)(LOCAL_LISTENER=LISTENER_EAST1)(REMOTE_LISTENER=REMOTE_EAST))'
 SCOPE=BOTH SID='db11g1';
####### Instance 2 #############
System altered.

ALTER SYSTEM
 SET LISTENER_NETWORKS='((NAME=NORTH_NET)(LOCAL_LISTENER=LISTENER_NORTH2)(REMOTE_LISTENER=global-scan.north.com:1521))',
                       '((NAME=SOUTH_NET)(LOCAL_LISTENER=LISTENER_SOUTH2)(REMOTE_LISTENER=REMOTE_SOUTH))',
                       '((NAME=EAST_NET)(LOCAL_LISTENER=LISTENER_EAST2)(REMOTE_LISTENER=REMOTE_EAST))'
SCOPE=BOTH SID='db11g2';

System altered.

####### Instance 3 #############
ALTER SYSTEM
 SET LISTENER_NETWORKS='((NAME=NORTH_NET)(LOCAL_LISTENER=LISTENER_NORTH3)(REMOTE_LISTENER=global-scan.north.com:1521))',
                       '((NAME=SOUTH_NET)(LOCAL_LISTENER=LISTENER_SOUTH3)(REMOTE_LISTENER=REMOTE_SOUTH))',
                       '((NAME=EAST_NET)(LOCAL_LISTENER=LISTENER_EAST3)(REMOTE_LISTENER=REMOTE_EAST))'
SCOPE=BOTH SID='db11g3';

System altered.

Checking if default database service db11g was registered proprely

#################### Network 1 #####################

lsnrctl status LISTENER
..
.
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.52)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.60)(PORT=1521)))
Services Summary...
..
.
Service "db11g" has 1 instance(s).
  Instance "db11g2", status READY, has 1 handler(s) for this service...
..
.
The command completed successfully

# In network 1 the remote_listener is registered in Listener SCAN
lsnrctl status LISTENER_SCAN3
..
.
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN3)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.62)(PORT=1521)))
Services Summary...
..
.
Service "db11g" has 3 instance(s).
  Instance "db11g1", status READY, has 2 handler(s) for this service...
  Instance "db11g2", status READY, has 2 handler(s) for this service...
  Instance "db11g3", status READY, has 2 handler(s) for this service...
..
.
The command completed successfully

#################### Network 2 #####################

lsnrctl status LISTENER_SOUTH
..
.
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SOUTH)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=140.120.120.101)(PORT=1522)))
Services Summary...
..
.
Service "db11g" has 3 instance(s).
  Instance "db11g1", status READY, has 1 handler(s) for this service...
  Instance "db11g2", status READY, has 2 handler(s) for this service...
  Instance "db11g3", status READY, has 1 handler(s) for this service...
The command completed successfully

#################### Network 2 #####################

lsnrctl status LISTENER_EAST
..
.
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_EAST)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=154.120.120.101)(PORT=1523)))
Services Summary...
..
.
Service "db11g" has 3 instance(s).
  Instance "db11g1", status READY, has 1 handler(s) for this service...
  Instance "db11g2", status READY, has 2 handler(s) for this service...
  Instance "db11g3", status READY, has 1 handler(s) for this service...
The command completed successfully

Creating Service “db11g_inbound”

Using “srvctl add service” with option -k it’s possible create a dependency specifying the number of network. However, this has no effect on listeners which is registered with the service, the service will be registered in all listener configured in RAC (LOCAL,REMOTE and NETWORK Listener). It rather serves only to ensure that the service is started, when the corresponding subnet, or the corresponding ora.netX.network resource is present and started.
Bad news, each service can be configure with  dependency  to only one network. We can not specify multiple networks numbers.

I created a workaround to fix it. (It’s not supported by Oracle)

Creating new Service DB11G_INBOUND

In example below a dependency will be created with ora.net2.network

srvctl add service -s db11g_inbound -m BASIC -e SELECT -d db11g -k 2 -r "db11g1,db11g2,db11g3"

Solution not supported by Oracle. Creating dependency of resource manually.

# Create service without specifying flag "-k" srvctl will use default network 1
srvctl add service -s db11g_inbound -m BASIC -e SELECT -d db11g  -r "db11g1,db11g2,db11g3"

# Getting current START and STOP Dependencies.

crsctl status resource ora.db11g.db11g_inbound.svc -f |grep -E 'START_DEPENDENCIES|STOP_DEPENDENCIES'
START_DEPENDENCIES=
hard(ora.db11g.db,type:ora.cluster_vip_net1.type)
weak(type:ora.listener.type)
pullup(type:ora.cluster_vip_net1.type)
pullup:always(ora.db11g.db)

STOP_DEPENDENCIES=hard(intermediate:ora.db11g.db,intermediate:type:ora.cluster_vip_net1.type)

# Modifying START Dependencies adding type cluster_vip_net (cluster_vip_net2 and cluster_vip_net3)
crsctl modify resource ora.db11g.db11g_inbound.svc -attr\
 "START_DEPENDENCIES='hard(\
ora.db11g.db,\
type:ora.cluster_vip_net1.type,\
type:ora.cluster_vip_net2.type,\
type:ora.cluster_vip_net3.type)\
weak(type:ora.listener.type) \
pullup(type:ora.cluster_vip_net1.type) \
pullup(type:ora.cluster_vip_net2.type) \
pullup(type:ora.cluster_vip_net3.type) \
pullup:always(ora.db11g.db)'"

# Modifying STOP Dependencies adding type cluster_vip_net (cluster_vip_net2 and cluster_vip_net3)
crsctl modify resource ora.db11g.db11g_inbound.svc -attr\
"STOP_DEPENDENCIES='hard(\
intermediate:ora.db11g.db,\
intermediate:type:ora.cluster_vip_net1.type,\
intermediate:type:ora.cluster_vip_net2.type,\
intermediate:type:ora.cluster_vip_net3.type)'"

# P.S The service db11g_inbound will not start if network (VIP IP) 1,2 and 3 is not avaliable, and you cannot stop network 1,2 and 3 if sevice db11g_inbound is online.
# Starting Service Created
srvctl start service -d db11g -s db11g_inbound

You must understand your environment and see if  is better to create a service for each network or create a service to all networks.
Remember that the service created always will be available in all networks (listeners), but in network maintenance may affect services if you manually created dependency.
Checking if Service was registered in all Listeners


############### Network 1 #########################
 lsnrctl status listener_scan1
..
.
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.63)(PORT=1521)))
..
.
Service "db11g_inbound" has 3 instance(s).
  Instance "db11g1", status READY, has 1 handler(s) for this service...
  Instance "db11g2", status READY, has 1 handler(s) for this service...
  Instance "db11g3", status READY, has 1 handler(s) for this service...
..
.

 lsnrctl status listener
..
.
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.60)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.217.52)(PORT=1521)))
..
.
Service "db11g_inbound" has 1 instance(s).
  Instance "db11g2", status READY, has 2 handler(s) for this service...
..
.
############### Network 2 #########################
lsnrctl status listener_south
..
.
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SOUTH)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=140.120.120.101)(PORT=1522)))
..
.
Service "db11g_inbound" has 3 instance(s).
  Instance "db11g1", status READY, has 1 handler(s) for this service...
  Instance "db11g2", status READY, has 2 handler(s) for this service...
  Instance "db11g3", status READY, has 1 handler(s) for this service...
..
.
############### Network 3 #########################
lsnrctl status listener_east
..
.
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_EAST)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=154.120.120.101)(PORT=1523)))
..
.
Service "db11g_inbound" has 3 instance(s).
  Instance "db11g1", status READY, has 1 handler(s) for this service...
  Instance "db11g2", status READY, has 2 handler(s) for this service...
  Instance "db11g3", status READY, has 1 handler(s) for this service...
..
.

Use the connect string above to connect  your clients on RAC Database 11.2


########### Connection String to Network 1 #####################
### Using SCAN ###########
db11g_inbound =
 (DESCRIPTION =
  (ADDRESS = (PROTOCOL = TCP)(HOST = global-scan.north.com)(PORT = 1521))
  (LOAD_BALANCE = YES)
  (CONNECT_DATA =
   (SERVER = DEDICATED)
   (SERVICE_NAME = db11g_inbound)
   (FAILOVER_MODE =
   (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5)
   )
  )

### Or Using VIP ###########
db11g_inbound =
 (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = northora01-vip.north.com)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = northora02-vip.north.com)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = northora03-vip.north.com)(PORT = 1521))
    )
  (LOAD_BALANCE = YES)
  (CONNECT_DATA =
   (SERVER = DEDICATED)
   (SERVICE_NAME = db11g_inbound)
   (FAILOVER_MODE =
   (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5)
   )
  )

########### Connection String to Network 2 #####################
db11g_inbound =
 (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = southora01-vip.north.com)(PORT = 1522))
      (ADDRESS = (PROTOCOL = TCP)(HOST = southora02-vip.north.com)(PORT = 1522))
      (ADDRESS = (PROTOCOL = TCP)(HOST = southora03-vip.north.com)(PORT = 1522))
    )
  (LOAD_BALANCE = YES)
  (CONNECT_DATA =
   (SERVER = DEDICATED)
   (SERVICE_NAME = db11g_inbound)
   (FAILOVER_MODE =
   (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5)
   )
  )

########### Connection String to Network 3 #####################
db11g_inbound =
 (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = eastora01-vip.north.com)(PORT = 1523))
      (ADDRESS = (PROTOCOL = TCP)(HOST = eastora02-vip.north.com)(PORT = 1523))
      (ADDRESS = (PROTOCOL = TCP)(HOST = eastora03-vip.north.com)(PORT = 1523))
    )
  (LOAD_BALANCE = YES)
  (CONNECT_DATA =
   (SERVER = DEDICATED)
   (SERVICE_NAME = db11g_inbound)
   (FAILOVER_MODE =
   (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5)
   )
  )

Useful docs about how configure multiple network an RAC env.

Data Guard: Redo Transport Services – How to use a separate network in a RAC environment. [ID 1210153.1] (11.1 or above)

How to Configure A Second Listener on a Separate Network in 11.2 Grid Infrastructure [ID 1063571.1] (11.2)

Data Guard Physical Standby 11.2 RAC Primary to RAC Standby using a second network [ID 1349977.1] (11.2)

Hope this helps.

Enjoy

Advertisements

17 Comments on “How configure Multiples Public Network in a Grid Infrastructure 11g R2 (11.2) environment”

  1. shivanand hindwan says:

    thank you much for this post .much Appreciated your efforts for sharing the knowledge ..ur all posts are really very helpful

    Best Regards
    Shivanand Hindwan

    Like

  2. […] work in one network the others network will provide services using only VIP hostnames. See it: How configure Multiples Public Network in a Grid Infrastructure 11g R2 (11.2) environment Regards, Reply With […]

    Like

  3. Jean-Yves says:

    Hi Levi,

    I’m a little confused about the Oracle statement “Listeners specified by the LISTENER_NETWORKS parameter should not be used in the LOCAL_LISTENER and REMOTE_LISTENER”. But they advise to include the default scan listener both in the init parameter REMOTE_LISTENER and in the REMOTE_LISTENER clause of the LISTENER_NETWORKS.

    Isn’t there some contradiction or did I miss something ?

    Like

    • Levi Pereira says:

      Hey Jean,

      This setup is exclusively for case that we use multiple network.

      LISTENER_NETWORKS act as route (ensuring that each connection will go to the correct destination), So when we are using different networks we must use this parameter to allow cross-registration.

      There is a restriction when we use LISTENER_NETWORKS. We cannot use SCAN feature over all network configured.

      SCAN feature (REMOTE_LISTENER and LOCAL_LISTENER) will work only in one public network, because scan is configured over one public network, as we can configure only one SCAN for entire Clusterware Stack and SCAN must be associated whith only one public network. So, only one public network we can use all feature of GI 11.2 the others public networks will be required use the VIP in our Oracle Clients like in version 11g R1 and 10g.

      About this question:
      But they advise to include the default scan listener both in the init parameter REMOTE_LISTENER and in the REMOTE_LISTENER clause of the LISTENER_NETWORKS?
      Who advised this? Where did you get this information?

      Regards,
      Levi Pereira

      Like

  4. Hey Levi,

    That was a very nice article.

    I’m facing a similar problem with a bit different scenario: I have two databases that run on the
    GI stack: North and South, both 3 instances. For security reasons, it has been decided that we need to separate the two databases to two different subnets.

    Your solution is excellent and gave me some ideas, but it’s referring to a single database. I was wondering how can we modify it so it will solve our scenario as well?

    Keep up the good work!
    Zohar

    Like

    • Levi Pereira says:

      Hi Zohar,
      This post is useful when you need configure multiple network in a Database.
      Far as I can understand you are configuring two network for two database, so each database in onw network.
      If it’s true, you don’t need to use parameter LISTENER_NETWORKS, because you are not using cross-registration between different network. You can use REMOTE and LOCAL Listener parameter pointing each to own Listener/Network.

      If I understood wrong, please correct me.

      e.g

      On Clusterware/RAC
      * net1 VIP/SCAN/Listener 1521
      * net2 VIP/Listener 1522

      * db1 (3 instances) (local_listner= listener(net1):1521/ remote_listener = scan_listener(net1):1521)
      * db2 (3 instances) (local_listener = listener(net2):1521/ remote_listener = tnsnames(local_listener(net2) from virtual IP node1,node2,node3))

      On Single Instance

      * net1 Listener 1521
      * net2 Listener 1522

      * db1 (local_listener = listener(net1):1521
      * db2 (local_listener = listener(net2):1522

      P.S You don’t need set remote_listener on single database, because does not existe remote instance on single.

      Like

  5. Domenico says:

    Fantastic!!! Thanks a lot

    Like

  6. Alessandro says:

    Great article! But i have another scenario:
    I have two nodes in a GI with two distinct database (ora1 and ora2 with each one shared home)
    I need to configure two scan listener on the same network (one for ora1 and 1 for ora2).
    It’s possible to do this ?

    Thanks
    Alessandro

    Like

    • Levi Pereira says:

      No, Is not supported to have two SCAN on same cluster.
      So, one database will support SCAN/VIP and the other database will support old fashion i.e using only VIP.

      Like

  7. martinfk says:

    the problem is only when you are using oracle rac- ie using scan listener and listener cross registartion correct?
    If I am on a single instance database (with no SCAN listener) and 2 local listeners, one for regular client connections and 2nd for data guard traffic I wouldnt have to specify”listener_networks”…is that a correct understanding?

    Like

  8. mlaitiMajdz says:

    Hello, I read carefully this article and I don’t understand what we can do with “south public”, “east public” addresses. Are these addresses usefull for something ?
    Best regards.

    Like

  9. Xiaoqiang Yao says:

    Hi, this is excellent article. I have implemented 2nd network. Now, we want to reverse the change. Say, I want to remove the newly added 2nd network. Can you shed somelight on it?
    I read the Oracle CRS manual in which I did not find srvctl add network or remove network options. Can share where do you find those ?

    Thank you very much.

    Xiaoqiang

    Like

    • Levi Pereira says:

      Hi Xiaoqiang,
      I’m sorry for the delayed response.

      You are right it’s not documented “srvctl remove network”, but the command exists.

      $ crsctl query crs activeversion
      Oracle Clusterware active version on the cluster is [11.2.0.3.0]
      $ srvctl remove network -h

      Removes a network configuration from the Oracle Clusterware.

      Usage: srvctl remove network {-k | -a} [-f] [-v]
      -k network number (default number is 1)
      -a All networks
      -f Force remove
      -h Print usage
      -v Verbose output

      I’ll contact Oracle Support to fix this issue.

      Hope this help.

      Like

  10. […] listen on the new network interface also. RAC needs special attention and it is described in this blog post. Basically you need to configure new set of VIP addresses, create a new network resource in […]

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s