Oracle 11gR2 2-node RAC on VMWare Workstation 8 – Part II

Add additional RPMs to support the Oracle database and grid installs.

Before we can install Oracle 11gR2 on our new VM, we need to load additional RPM packages that are required by the Oracle database and grid. Oracle publishes a list of the required VMs broken down by the supported Linux distributions at the following URL:
RPMs Required for Oracle 11gR2 on RHEL 5

As we can see from this list, the required RPMs for Red Hat Enterprise Linux 5 are as follows:

binutils-2.17.50.0.6
compat-libstdc++-33-3.2.3
compat-libstdc++-33-3.2.3 (32 bit)
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
gcc-4.1.2
gcc-c++-4.1.2
glibc-2.5-24
glibc-2.5-24 (32 bit)
glibc-common-2.5
glibc-devel-2.5
glibc-devel-2.5 (32 bit)
glibc-headers-2.5
ksh-20060214
libaio-0.3.106
libaio-0.3.106 (32 bit)
libaio-devel-0.3.106
libaio-devel-0.3.106 (32 bit)
libgcc-4.1.2
libgcc-4.1.2 (32 bit)
libstdc++-4.1.2
libstdc++-4.1.2 (32 bit)
libstdc++-devel 4.1.2
make-3.81
sysstat-7.0.2

As it turns out, most of these are already loaded by the standard install we just completed in Part I. We only actually need to load the following RPMs:

compat-libstdc++-33-3.2.3
compat-libstdc++-33-3.2.3 (32 bit)
elfutils-libelf-devel-0.125
glibc-devel-2.5
libaio-devel-0.3.106
libaio-devel-0.3.106 (32 bit)
sysstat-7.0.2
unixODBC-2.2.11
unixODBC-2.2.11 (32 bit)
unixODBC-devel-2.2.11
unixODBC-devel-2.2.11 (32 bit)

Eagle eyed Super Scouts may notice something odd about our distilled list. There are several RPMs that we need here that are NOT listed on the official list from Oracle. Well like the tag line of this blog states – “when you just want this crap to work”. This second and distilled list is the real list of additional RPMs you need to install Oracle on the RHEL 5.5 VM.
To install the required RPMs we need to re-insert the Red Hat Enterprise Linux 5.5 install media. In my example I am using an ISO of the DVD which makes matters very simple. If you have CD based media you may to go hunting through the discs for the RPMs.
First, use the VM -> Removable Devices -> CD/DVD (ISO) menu option in VMWare Workstation to re-mount the ISO of the install media.

Once mounted, Linux should open the media on the root desktop.

We now need a terminal session to navigate to the directory on the install media where we will find the RPMs we need.

[root@localhost Server]# cd "/media/RHEL_5.5 x86_64 DVD/Server"
[root@localhost Server]# pwd
 /media/RHEL_5.5 x86_64 DVD/Server

Now the following single-line command can be used to load every missing RPM in a single step:
rpm -ivh compat-libstdc++-33-3.2.3-61.*.rpm glibc-devel-2.5-49.i386.rpm libaio-devel-0.3.106-5.*.rpm sysstat-7.0.2-3.el5.x86_64.rpm unixODBC-2.2.11-7.1.*.rpm unixODBC-devel-2.2.11-7.1.*.rpm elfutils-libelf-devel*.x86_64.rpm

If successful, Linux will respond as follows:

[root@localhost Server]# rpm -ivh compat-libstdc++-33-3.2.3-61.*.rpm glibc-devel-2.5-49.i386.rpm libaio-devel-0.3.106-5.*.rpm sysstat-7.0.2-3.el5.x86_64.rpm unixODBC-2.2.11-7.1.*.rpm unixODBC-devel-2.2.11-7.1.*.rpm elfutils-libelf-devel*.x86_64.rpm
 warning: compat-libstdc++-33-3.2.3-61.i386.rpm: Header V3 DSA signature: NOKEY, key ID 37017186
 Preparing... ########################################### [100%]
 1:unixODBC ########################################### [ 8%]
 2:sysstat ########################################### [ 17%]
 3:compat-libstdc++-33 ########################################### [ 25%]
 4:unixODBC ########################################### [ 33%]
 5:compat-libstdc++-33 ########################################### [ 42%]
 6:unixODBC-devel ########################################### [ 50%]
 7:unixODBC-devel ########################################### [ 58%]
 8:libaio-devel ########################################### [ 67%]
 9:libaio-devel ########################################### [ 75%]
 10:glibc-devel ########################################### [ 83%]
 11:elfutils-libelf-devel-s########################################### [ 92%]
 12:elfutils-libelf-devel ########################################### [100%]

I am going to advocate you install one more very important RPM now. This is not listed as an Oracle requirement, but it will enable you to stand up your own private DNS server on your RAC nodes.
With 11gR2 Oracle has introduced a SCAN address for RAC clusters. SCAN is an acronym for Single Cluster Address Name. It provides an extremely crude round-robin method of trying cluster addresses in turn. Furthermore, the new OUI installers demand SCAN be enabled and configured otherwise they complain loudly.
Whereas you can work-around not having a DNS server, the more elegant solution is to stand up one, and luckily Red Hat makes this pretty simple. So let’s grab the RPM we need now and we will configure it later.

[root@localhost Server]# rpm -ivh bind-9.3.6-4.P1.el5_4.2.x86_64.rpm
warning: bind-9.3.6-4.P1.el5_4.2.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 37017186
Preparing...                ########################################### [100%]
   1:bind

Once we are done with this step we can eject the install media.
You might consider taking a fresh clone of your VM now since you have all RPMs loaded for any future Oracle 11gR2 installations you might want to do.

Configure the Linux kernel to support Oracle 11gR2.

 Before we can install Oracle 11gR2 on our new VM, we need configure the Linux kernel. The following steps modify key settings to allow Oracle to execute. These steps are taken from the Oracle install guide located at the following URL:
Linux Kernel Parameters for Oracle 11gR2 on RHEL 5

Edit the /etc/sysctl.conf and add following lines:

# added for Oracle 11gR2
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

Now execute “sysctl -p” command to apply the new settings:

[root@localhost ~]# /sbin/sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

Edit the /etc/pam.d/login file and add following line:

# added for Oracle 11gR2
session    required     pam_limits.so

Edit the /etc/security/limits.conf file and add following lines:

# added for Oracle 11gR2
oracle    soft  nproc   2047
oracle    hard  nproc   16384
oracle    soft  nofile  1024
oracle    hard  nofile  65536

Check current status of SELinux:

[root@localhost ~]# /usr/sbin/getenforce
Enforcing

If output is Enforcing then change mode to Permissive as follows:

[root@localhost ~]# /usr/sbin/setenforce 0

To make the change permanent, modify the /etc/sysconfig/selinux change value of SELINUX variable to disabled:

[root@localhost ~]# cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing - SELinux security policy is enforced.
#       permissive - SELinux prints warnings instead of enforcing.
#       disabled - SELinux is fully disabled.
SELINUX=disabled
# SELINUXTYPE= type of policy in use. Possible values are:
#       targeted - Only targeted network daemons are protected.
#       strict - Full SELinux protection.
SELINUXTYPE=targeted

Now we are going to disable the Linux firewall, which will otherwise interfere with the RAC install. First let’s make sure the firewall service is stopped:

[root@localhost ~]# service iptables stop
Flushing firewall rules: [  OK  ]
Setting chains to policy ACCEPT: filter [  OK  ]
Unloading iptables modules: [  OK  ]

To permanently disable the firewall we will disable the service:

[root@localhost ~]# chkconfig iptables off

Now we have the Linux kernel ready for Oracle.

Create the Oracle user account, OS groups and mount point.

 The Oracle software will be installed and owned by the oracle user account. In additional several OS groups are created that allow other non oracle OS users privileged access to the database and grid resources.
Login as root and create the user oracle account and the OS groups:

[root@localhost ~]# groupadd dba
[root@localhost ~]# groupadd oinstall
[root@localhost ~]# groupadd asmdba
[root@localhost ~]# groupadd asmadmin
[root@localhost ~]# useradd -g oinstall -G dba,asmdba,asmadmin oracle

Now set the password for the oracle user. Since this is development, I suggest we stick with “oracle”.

[root@localhost ~]# passwd oracle
Changing password for user oracle.
New UNIX password: 
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password: 
passwd: all authentication tokens updated successfully.

Now we create a directory into which the oracle software will be installed. We need to set this new directory to be owned by oracle:

[root@localhost ~]# mkdir /u01
[root@localhost ~]# mkdir /u01/app
[root@localhost ~]# chown oracle:dba /u01/app

Now we can log into our new oracle account and configure the bash profile:

[root@localhost ~]#  su - oracle
[oracle@localhost ~]$  vi ~/.bash_profile

Add the following lines to the profile script:

TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_HOSTNAME=`hostname`; export ORACLE_HOSTNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=/usr/sbin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -p 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

Our VM now has all the necessary RPMs, Kernel settings and Oracle account privileges to load Oracle 11gR2.
You might consider another clone of your VM at this point, or at least a snapshot to fall back to in the event of problems.