Oracle 11gR2 (11.2.0.2) Installation - Database software

This is the 2nd in my series of Oracle11gR2 installations, focusing now on the installation of the database or RDBMS software for 11.2.0.2 having already installed the Grid Infrastructure component per my previous post here. If you don't want to install the Grid Infrastructure and use ASM (and Oracle Restart) that is fine, go ahead and skip that first article. In such a case I assume you would be using file systems (perhaps with dNFS which is a post for another time).

Here again, I will be using the command line approach since this is an easy way to have everything scripted and automated (and not require a GUI). I'll show the parameters that need to be adjusted, but if you are not comfortable then I'd suggest doing an initial GUI installation, saving that response file when prompted and then using the saved response file as your gold image for further scripted installations. Note that in the below I use RDBMS_HOME instead of ORACLE_HOME to make the distinction between the actual database home and the grid infrastructure home.

Requirements

See my previous post here

Installation

1. If this is your first installation, then you will want to create the '/etc/oraInst.loc' file, as the root user:

Note: This is optional since you will be prompted at the end of the installation to run $ORACLE_BASE/../oraInventory/orainstRoot.sh which does this for you if this files does not exist.

echo "inventory_loc=/oracle/app/oraInventory

inst_group=dba"

> /etc/oraInst.loc

chown oracle:dba /etc/oraInst.loc

chmod 664 oraInst.loc

2. Edit the DB response file, 'db_inst.rsp' for the values as show below:

ORACLE_HOSTNAME=oradb01

ORACLE_BASE=/oracle/app/oracle

ORACLE_HOME=/oracle/app/product/11.2.0/db_1

3. Run the DB installation (using responsefile and silent installation) as the oracle user:

Note: Ensure you set your DISPLAY environment variable, or you are at run level 5, otherwise you will encounter an error.

./runInstaller -silent -noconfig -responseFile /home/oracle/rsp/db_inst.rsp

4. As the root user run '$RDBMS_HOME/root.sh' when the above completes as prompted.

5. As the oracle user, create an OCM response file. This saves a lot of time later down when you are prompted for those values. Simple run the following and follow the instructions to create and save the response file:

$RDBMS_HOME/OPatch/ocm/bin/emocmrsp

6. Apply the latest OPatch patch (MOS patch ID 6880880), then the latest PSU to this installation as the oracle user. Simple unzip the OPatch patch to the $RDBMS_HOME. For the PSU, unzip to a temporary location, navigate to the patch location and run:

$RDBMS_HOME/OPatch/opatch apply -ocmrf /home/oracle/rsp/ocm.rsp 

7. Apply patch 12431716 (as required by PSU 2) by unzipping to a temporary location, change to the patch directory and as oracle user running: 

$RDBMS_HOME/OPatch/opatch apply -ocmrf /home/oracle/rsp/ocm.rsp

At this point you have fully prepared GI and RDBMS software installations with a ready listener and two disk groups. Now you can create a new database, or migrate an exiting database. I'll leave the new installation to another post in which I'll show how to use DBCA and a template to do it silently, without a GUI.

Oracle Exadata Storage Expansion Rack

Today Oracle announced the availability of the Exadata Storage Expansion Rack and I can't say I didn't see this one coming. I don't think my current company was alone in forecasting running low on space within the confines of a single Exadata machine, but not wanting to pay for another full rack of Exadata since the compute nodes were not needed. Sure, you can buy just the additional storage, but it's a pain as from what I've seen that is a negotiation with Oracle again.

There does not seem to be anything different between previously getting just the additional Storage Servers vs. now buying a Storage Expansion Rack. I see this announcement as just Oracle making official or public the fact that you can get the Storage Servers without the compute nodes. As can be expected, the racks are available in the same configurations as the database machine itself, i.e. quarter, half, and full rack, and include IB switches and cables. Since these are the standard configurations, it is a bit curious as to why Oracle choose to nonetheless license the Storage Software separately. Everyone knows the Storage Software costs, and the configuration is fixed so why bother? Sure, you can negotiate discounts for the software costs (as everyone does), but why not just give the full starting price without asking me to go calculate it?

• Full rack: 18 Oracle Exadata Storage Servers with up to 6.75 TB of smart flash cache, 432 TB of raw disk storage and 216 CPU cores, will run you $750,000 a box.
• Half rack: 9 Exadata Storage Servers has 3.4 TB of smart flash cache, 216 TB of raw disk storage and 108 CPUs will run you $425,000.
• Quarter rack: system has 1.5 TB of flash cache, 96 TB of disk storage and 48 CPU cores costs $225,000.

Another point of note is that this is really for backup, storage archival, or low bandwidth processing as only the high capacity or HC disks (7.2K rpm, 2TB) are used in the current expansion racks. Sorry, no high performance (HP) or 15K rpm SAS disks (600GB) yet. Considering this is an expansion rack to augment storage and not necessarily for primary usage this is not to be totally unexpected though it would be nice to have that option also available in case you are in the situation of requiring additional HP storage. Yes, you still get the TB of Smart Flash Cache and the Storage Software but it's still not as fast when you need 15K rpm and true SAS. Maybe this might be a future standard offering as well with enough customer demand.

If you want to read more you can head over to the Oracle Exadata website which has the full details, a direct link to the PDF data sheet is here as well. But happy times are ahead. My prediction is that we will also see some sort of Exadata announcement at OpenWorld 2011 (where's my blogger approval Oracle?) perhaps newer Intel CPUs and faster/more storage and/or a MySQL machine.

Oracle 11gR2 (11.2.0.2) Installation - Grid Infrastructure

I quite often find myself doing Oracle installations and referencing back to the manuals, read me, other blogs, and various other information sources in an attempt to ensure I get everything just perfect. This is regardless of having done it many times and having documented it myself. These days however, I find this a time consuming exercise, especially given my increasing workload so I thought to simply blog about my method which follows below.

The installation below utilizes a command line only method as I often need to be do multiple installations in a standard, and scripted way. The best method I've found for reaching this point is to initially start with a GUI approach, from which the response file is saved which can then be used for further silent, command line driven installations. I'll start with the installation of the Grid Infrastructure component which is used for ASM and Oracle Restart, and continue with a follow-up posting with the RDBMS software installation. In the method that follows I use a simplified setup with only a single group assigned to the oracle user, i.e. 'dba'. Of course you will need to modify the below for your environment.

Requirements

OS: SUSE Linux Enterprise Service 10.3 (64-bit)

Packages (standard installation):

make-3.80-202.2
binutils-2.16.91.0.5-23.34.33
libstdc++33-3.3.3-7.8.1
gcc-4.1.2_20070115-0.29.6
gcc-c++-4.1.2_20070115-0.29.6
glibc-2.4-31.74.1
glibc-devel-2.4-31.74.1
glibc-devel-32bit-2.4-31.74.1
ksh-93s-59.11.35
libaio-0.3.104-14.2
libaio-32bit-0.3.104-14.2
libaio-devel-0.3.104-14.2
libaio-devel-32bit-0.3.104-14.2
libelf-0.8.5-47.2
libgcc-4.1.2_20070115-0.29.6
libstdc++-4.1.2_20070115-0.29.6
libstdc++-devel-4.1.2_20070115-0.29.6
numactl-0.9.6-3.25.28
sysstat-8.0.4-1.7.27
unixODBC-2.2.12
unixODBC-devel-2.2.12
unixODBC-32bit-2.2.12
oracleasmlib-2.0.4-1.SLE10
oracleasm-2.0.5-7.4.50
oracleasm-support-2.1.4-1.SLE10

Configuration files & settings:

/etc/sysctl.conf

# Oracle settings
kernel.core_uses_pid = 1
kernel.suid_dumpable = 1
kernel.msgmni = 2878
# physical RAM size (8 GB) / pagesize (4096)
kernel.shmall = 2097152
# 1/2 of physical RAM typically, 5 GB here
kernel.shmmax = 5368709120
kernel.shmmni = 4096
#kernel.sem = 250 32000 100 128
kernel.sem = 250 32000 100 142
# 512 x processes (for example 6815744 for 13312 processes)
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.ipv4.conf.default.rp_filter = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
# For RDS use larger value below
#net.core.wmem_max = 2097152
net.core.wmem_max = 1048576
# group ID of dba group
vm.hugetlb_shm_group = 7001
# number of hugepages (2 MB size) to be used for ASMM
vm.nr_hugepages=2560

/etc/security/limits.conf

# Oracle settings
oracle soft core unlimited
oracle hard core unlimited
oracle soft nproc 131072
oracle hard nproc 131072
oracle soft nofile 131072
oracle hard nofile 131072
oracle soft stack 32768
oracle hard stack 32768
oracle soft memlock 5242880
oracle hard memlock 5242880
root soft core unlimited
root hard core unlimited
root soft nproc 131072
root hard nproc 131072
root soft nofile 131072
root hard nofile 131072
root soft memlock 5242880
root hard memlock 5242880

/etc/pam.d/su

# Oracle setting
session  required       pam_limits.so

/etc/pam.d/xdm

# Oracle setting
session  required       pam_limits.so

/etc/sysconfig/ntp
NTPD_OPTIONS="-x -u ntp"    # add '-x' to existing string

~/.ssh/config
Host *
ForwardX11 no    # Set in the oracle user's file

~/.bashrc
if [ -t 0 ]; then
   stty intr ^C
fi    # Disable terminal output commands (assumes bash shell)

Links

ln -s /bin/fuser /sbin/fuser
ln -s /usr/bin/ssh /usr/local/bin/ssh
ln -s /usr/bin/perl /usr/local/bin/perl
ln -s /usr/bin/scp /usr/local/bin/scp

Installation

1. (optional) Create '/etc/oraInst.loc' file as the root user.

Note: This step is optional as it will be done automatically for first time installations when the oraInst.sh script is run.

echo "inventory_loc=/oracle/app/oraInventory

inst_group=dba"

> /etc/oraInst.loc

chown oracle:dba /etc/oraInst.loc

chmod 664 oraInst.loc

2. Edit GI response file, 'grid.rsp' for the below values:

ORACLE_HOSTNAME=oradb01

ORACLE_BASE=/oracle/app/oracle

ORACLE_HOME=/oracle/app/11.2.0/grid_1

3. As the oracle user, run GI installation (using responsefile 'grid.rsp' and silent installation).

Notes: 
   - Ensure your DISPLAY environment variable is set, or you are at run level 5. Otherwise you will not be able to run the command successfully.
   - You may see some warnings for the installation pertaining to a missing library package compat-libstdc++ which can safely be ignored as for SLES 10.3 and greater libstdc++33 is used instead (reference MOS: 'Requirements for Installing Oracle 11gR2 64-bit (AMD64/EM64T) on SLES 10 (Doc ID 884435.1)').

./runInstaller -silent -noconfig -responseFile /home/oracle/rsp/grid.rsp

4. As the root user, run '/oracle/app/oraInventory/oraInst.sh', '$GI_HOME/root.sh' and then '$GI_HOME/crs/install/roothas.pl' (per log file contents) after the above completes. See below for example run of '$GI_HOME/crs/install/roothas.pl':

/oracle/app/11.2.0/grid_1/perl/bin/perl -I/oracle/app/11.2.0/grid_1/perl/lib -I/oracle/app/11.2.0/grid_1/crs/install /oracle/app/11.2.0/grid_1/crs/install/roothas.pl

Using configuration parameter file: /oracle/app/11.2.0/grid_1/crs/install/crsconfig_params

Creating trace directory

LOCAL ADD MODE

Creating OCR keys for user 'oracle', privgrp 'dba'..

Operation successful.

LOCAL ONLY MODE

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

CRS-4664: Node oemgc02 successfully pinned.

Adding daemon to inittab

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies - this may take some time.

FATAL: Module oracleoks not found.

FATAL: Module oracleadvm not found.

FATAL: Module oracleacfs not found.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9121: Failed to detect control device '/dev/asm/.asm_ctl_spec'.

ACFS-9310: ADVM/ACFS installation failed.

ACFS-9311: not all components were detected after the installation.

oemgc02     2011/06/23 10:02:19     /oracle/app/11.2.0/grid_1/cdata/oemgc02/backup_20110623_100219.olr

Successfully configured Oracle Grid Infrastructure for a Standalone Server

4b. To address the failure of the modules as seen above, reference MOS article "ACFS-9327 ACFS-9121 ACFS-9310: ADVM/ACFS installation failed [ID 1265276.1]" and run the following as the root user:

mkdir -p /lib/modules/`uname -r`/extra/usm

cp -p /lib/modules/2.6.16.60-0.54.5*/extra/usm/* /lib/modules/`uname -r`/extra/usm

/sbin/depmod

4c. OHASD will not starting after reboot on SLES (reference MOS: "OHASD not Starting After Reboot on SLES [ID 1325718.1]"). Run the following to fix this issue:

oradb01:~ # chkconfig --list|grep ohasd

init.ohasd                0:off  1:off  2:off  3:off  4:off  5:off  6:off

ohasd                     0:off  1:off  2:off  3:off  4:off  5:off  6:off

oradb01:~ # chkconfig --list|grep raw

raw                       0:off  1:off  2:off  3:off  4:off  5:off  6:off

oradb01:~ # chkconfig raw on

oradb01:~ # chkconfig ohasd on

oradb01:~ # chkconfig --list|grep raw

raw                       0:off  1:off  2:on   3:on   4:off  5:on   6:off

oradb01:~ # chkconfig --list|grep ohasd

init.ohasd                0:off  1:off  2:off  3:off  4:off  5:off  6:off

ohasd                     0:off  1:off  2:off  3:on   4:off  5:on   6:off

Note: Just turning on OHASD will result in error:

oradb01:~ # chkconfig ohasd on

insserv: Service raw has to be enabled for service oracle_has

insserv: exiting now!

/sbin/insserv failed, exit code 1

5. Apply latest OPatch patch, then latest PSU to GI software installation (Oracle Restart Home). As root user run '$GI_HOME/OPatch/opatch auto /oracle/stage/patches_grid -oh $GI_HOME -ocmrf /home/oracle/rsp/ocm.rsp'. An example below:

oemgc02:/oracle/stage/patches_grid # $ORACLE_HOME/OPatch/opatch auto /oracle/stage/patches_grid -oh $ORACLE_HOME -ocmrf /home/oracle/rsp/ocm.rsp

Executing /usr/bin/perl /oracle/app/11.2.0/grid_1/OPatch/crs/patch112.pl -patchdir /oracle/stage -patchn patches_grid -oh /oracle/app/11.2.0/grid_1 -ocmrf /home/oracle/rsp/ocm.rsp -paramfile /oracle/app/11.2.0/grid_1/crs/install/crsconfig_params

opatch auto log file location is /oracle/app/11.2.0/grid_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-06-23_10-32-00.log

Detected Oracle Restart install

Using configuration parameter file: /oracle/app/11.2.0/grid_1/crs/install/crsconfig_params

Successfully unlock /oracle/app/11.2.0/grid_1

patch /oracle/stage/patches_grid/12311357  apply successful for home  /oracle/app/11.2.0/grid_1

patch /oracle/stage/patches_grid/11724916  apply successful for home  /oracle/app/11.2.0/grid_1

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9312: Existing ADVM/ACFS installation detected.

ACFS-9314: Removing previous ADVM/ACFS installation.

ACFS-9315: Previous ADVM/ACFS components successfully removed.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies - this may take some time.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9309: ADVM/ACFS installation correctness verified.

CRS-4123: Oracle High Availability Services has been started.

5b. Apply patch 12431716, as root user run '$GI_HOME/crs/install/roothas.pl -unlock'. An example below:

oemgc02:/oracle/stage/12431716 # $ORACLE_HOME/crs/install/roothas.pl -unlock

Using configuration parameter file: /oracle/app/11.2.0/grid_1/crs/install/crsconfig_params

Successfully unlock /oracle/app/11.2.0/grid_1

Next, as GI owner run '$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local  -ocmrf /home/oracle/rsp/ocm.rsp'.

Next, as root user run '$GI_HOME/crs/install/roothas.pl -patch'. An example below:

oemgc02:/oracle/stage/12431716 # $ORACLE_HOME/crs/install/roothas.pl -patch

Using configuration parameter file: /oracle/app/11.2.0/grid_1/crs/install/crsconfig_params

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9312: Existing ADVM/ACFS installation detected.

ACFS-9314: Removing previous ADVM/ACFS installation.

ACFS-9315: Previous ADVM/ACFS components successfully removed.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies - this may take some time.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9309: ADVM/ACFS installation correctness verified.

CRS-4123: Oracle High Availability Services has been started.

6. Create listener from GI home (it will automatically get added to CRS by NETCA). By creating the listener before the ASM or database instances, Clusterware will automatically make the dependency relation. As oracle user edit the netca.rsp file and run:

Note: Ensure you have set the DISPLAY environment variable, or you are at run level 5. Otherwise the command will fail.

$GI_HOME/bin/netca -silent -responsefile /home/oracle/resp/netca.rsp

7. Create an ASM instance in silent mode by using ASMCA. The parameters for ASM in 11.2 require a disk group. I like to create a separate disk group (using a 2 GB disk) for this purpose which also serves the purpose of storing the OCR and voting files should the instance be migrated to RAC (since I also like to store those files using a separate disk group). To create this initial disk group, called SYSTEMDG run ASMCA as the oracle (or GI owner) user, then to create the additional disk groups for DATA and FRA, run similar commands using '-createDiskGroup' instead of '-configureASM'. See the examples below:

oracle@oradb01:~/bin> $GI_HOME/bin/asmca -silent -configureASM -sysAsmPassword sysPassw0rd -asmsnmpPassword asmsnmpPassw0rd -diskGroupName SYSTEMDG -diskList 'ORCL:SYSTEMD' -redundancy EXTERNAL -au_size 4 -compatible.asm '11.2.0.2.0' -compatible.rdbms '11.2.0.2.0'

ASM created and started successfully.

Disk Group SYSTEMDG created successfully.

oracle@oradb01:~/bin> $GI_HOME/bin/asmca -silent -createDiskGroup -sysAsmPassword sysPassw0rd -diskGroupName D1_T2_A4 -diskList 'ORCL:D_T2_*' -redundancy EXTERNAL -au_size 4 -compatible.asm '11.2.0.2.0' -compatible.rdbms '11.2.0.2.0'

Disk Group D1_T2_A4 created successfully.

oracle@oradb01:~/bin> $GI_HOME/bin/asmca -silent -createDiskGroup -sysAsmPassword sysPassw0rd -diskGroupName F1_T2_A4 -diskList 'ORCL:F_T2_*' -redundancy EXTERNAL -au_size 4 -compatible.asm '11.2.0.2.0' -compatible.rdbms '11.2.0.2.0'

Disk Group F1_T2_A4 created successfully.


You have now successfully installed Grid Infrastructure 11.2.0.2, applied a PSU and recommended patch, and created an ASM instance (along with a corresponding listener). At this point you are ready to commence the database installation process which I will include in a follow-up blog.