Friday, October 21, 2011

Enterprise Manager 12c Cloud Control Installation

This is a long overdue posting on my experience with installing the recently released Oracle Enterprise Manager 12c (12.1.0.1). But before a few items of interest that I think should be pointed out:

Product Rename - Oracle has renamed the product from "Enterprise Manager Grid Control" to "Enterprise Manager Base Platform". A side effect of this is easier knowledge searching and support for this specific release/product.

Architecture redesign - The architecture has been redesigned (a good thing) to utilize plug-ins for everything. So now the OMS can be patched and/or upgraded independent of any capabilities in monitored targets, and even more importantly, so plug-ins can now be patched and/or upgraded to take advantage of new features and/or functionalities of their respective targets. For example, when Database 12c is released a simple upgrade to the "Enterprise Manager for Oracle Database" plugin should be all that is required to allow EM12c to monitor and leverage all the new features DB12c would offer. Previously, the Agent and the OMS would require patches and/or upgrades to fully support such a new database release. At least this is the plan, we will see when DB12c is released if this is indeed the case.

Repository is master - The repository is now the master for data with the Agents being even more stateless. This is an improvement which allows agents to simple pass monitoring data to the OMS, and hence can be upgraded or re-installed without any loss of target data since they can be fully re-synchronized from the repository.

On now with my installation...

Installation Setup
I wont go into the full detailed list of prerequisites since you can get that from the documentation here. It is pretty much the usual, you can simple reuse your settings for database (11.2 since that is what I use) installations. Lets just say it was all accurate as at this writing. So I created a VM guest environment with the following specifications:

OS: RHEL 5.6 x86_64
RAM: 8 GB
CPU: 2 x 2.67 GHz Intel Xeon
VM: Yes (VMware 4.1u1)

While many may shy aware from using VMs, I think EM is one area where this is ideail since there are no license implications (except for the OMR, more on that later) and the requirements are optimal for a VM. For production deployment I would have at least two OMS, spread across separate cluster hosts with node affinity setup so both OMS never share the same host (unless temporarily necessary when doing some form of VM host maintenance). There are multiple ways to do the configuration, if you need ideas based on your unique situation please feel free to contact me.

On the issue if license, Enterprise Manager does come with a restricted use license for all its components (the basic tool is free, you pay for the plugins, connectors, and packs). This means however, that if you chose to go outside those boundaries, you will need to license the component as appropriate. For example, the repository database is licensed as an Enterprise Edition Single Instance, choosing to use a RAC setup would mean you need to pay for those RAC licenses, as well as the second instance (or each additional RAC instance as the case may be). This is all documented in the license information here.

As most should be aware since EM 11.1, the OMR is no longer installed as part of the installation. So you will need to install and setup an existing database to use as your OMR prior to installation of EM. For my installation I chose 11.2.0.3. Yes, as of this writing it is not yet on the certification listing, but this minor and I fully expect this version to be included shortly though for production I will be adhering to the certification matrix. I chose to let the installation install WebLogic (10.3.5). I prefer this method since it is simply cleaner and avoids any unnecessary patch, or upgrade issues which would result from using a shared WebLogic farm.

Step 1: Meet pre-requisite requirements
Check the documentation here for the requirements.


Step 2: Setup the repository database
This is included in the documentation here but I chose to include it since a few things have changed, or perhaps have not yet been documented, and otherwise stand out. In prior versions EM required setting '_b_tree_bitmap_plans=false', and 'aq_tm_processes=1' but there is no mention of these settings being required for EM12c (so I did not include them). Also, the documentation states "The instance parameters db_cache_size, large_pool_size, java_pool_size, and streams_pool_size must not be set for Enterprise Manager 12c installations.". I definitily do not agree with this note as my experience has been not setting minimum values for these parameters in the real world causes headaches with pool starvation and the reallocation not keeping up with demand (i.e. poor performance). AMM is recommended but since I'm installing on Linux and prefer to enable huge pages (in my humble experience this provides better performance), I instead use ASMM with huge pages (huge pages and AMM are mutually exclusive). I will leave my thoughts of ASMM and AMM for another blog. So in any case, the installation will complain (give a warning) but will go through otherwise with no issues. My usage thus far has also not encountered any issues. A final thing to note, that is not mentioned, is that the automatic stats gathering job should be disabled in the repository database. This will be detected during installation and can be disabled at that point so it is not critical.

So after you download, met all the pre-requisits and built your repository you are ready for the actual installation.


Step 3: Start installation
./runInstaller

em12c-install_01

This is my lab environment which has no internet access so I disable the email option and clicked 'Next'. This leads to a follow-up screen confirming your intent to remain uninformed:

em12c-install_01b

Step 4: Software updates

em12c-install_02

I selected to skip the updates (not that any are available just yet) since I have no internet access.

Step 5: Prerequisite Checks

em12c-install_03

Ensure that all the checks are successful here otherwise you may run into problems later on. I've been a good boy and everything is passed successfully.


Step 6: Select your installation type

em12c-install_04

Even though this a lab environment I prefer to choose the 'Advanced' option to 'Create a new Enterprise Manager System' since this is what I will be doing when testing for my company. So I then will need to specify a location for the middleware software location.

em12c-install_04b

You can also choose to upgrade, or install just the software. I'll let you know how things work out with the other options once I've had a chance to test.


Step 7: Select Plug-Ins

em12c-install_05

Here is where things get interesting. This is the new architecture where everything is a plug-in. Those you see in the screen shot are selected by default and mandatory (i.e. you can't unselect). What is not so obvious is that the 'Oracle Fusion Middleware' plug-in is dependent on 'Exalogic Elastic Cloud Infrastructure', which is of course not selected. Seems silly to not just select and install the two together since there is a dependency, even if it is a soft one since you can move on by selecting to install it later. I chose to install it now and avoid any potential issues (admittedly I doubt there would be since I would not be using the feature).

em12c-install_05b



Step 8: Weblogic Server Configuration

em12c-install_06

Default domain is GCDomain and default node manager is 'nodemanager'. This could be changed when invoking the runInstaller but I don't see any reason. Enter the passwords, and the OMS instance base location.


Step 9: Repository Database Configuration Details

em12c-install_07

Clicking 'Next' makes a test connection, and if successful verification of the repository commences. I did not know of the requirement to disable the automatic stats job in the repository database, but the installion handles disabling it (if I so choose which I did):

em12c-install_07b

After fixing the stats collection job, a further warning is given for failing to exactly meet requirements:

em12c-install_07c

My settings are higher than the requirement so most of these are silly and should not be displayed as they could cause unnecessary changes. For example the minimum for 'processes' is 300, and mine is set to 1024. Annoying me about the minimum which I've exceeded is a bug IMHO, after all it is not a maximum recommendation.

Next up is the SYSMAN credentials and tablespace setup. Enter the appropriate file system location for the database files. Here the system has detected (partially) that I'm using ASM, so I oblige by entering my disk group information to store the tablespace data files.

em12c-install_08



Step 9: Port Configuration

em12c-install_09

I don't have any need to modify anything here and leave the defaults. Your security requirements may be different and require the usage of non-standard ports, which is fine.


Step 10: Review

em12c-install_10
Almost done!

Step 11: Install

em12c-install_11

Watch (or get some refreshments) as the installation should breeze through! Note that EM12c uses the same database 11g process of packing already installed software, so the installation is essentially compressed into copying over those files, relinking for your system, and making some configuration changes.

Run the 'allroot.sh' script when prompted:

em12c-install_12

Step 12: Installation Complete!

em12c-install_13

All should have gone well so you need only review the results of your successfully installation, and make some notes, if you don't want to check files later, as to URLs and ports. I highly recommend following the note about exporting the configuration which includes the encryption key. This should actually be a scheduled job and become part of your backup/recovery process for EM.


Step 13: Access the New UI

em12c-console_01
Redesigned (dear I say gorgeous?) login prompt.

em12c-console_02
Your first access greets you with an updated license acceptance screen. Following which you will need to select your future welcome screen:

em12c-console_03
This can be changed later, but the point is EM12c allows your to easily customize your welcome screen. Very cool!


So that is it! It is a very clean and easy installation process, compared with previous versions, and I am left impressed. Having used it for a little while things do take a bit of getting used to and finding your way around (as with any redesign), but it is a definte improvement. I'll be following up with usage posts in future and letting you know what I find.

Monday, October 03, 2011

Oracle Enterprise Manager 12c Grid Control

I've always used Enterprise Manager to some extent. I can remember when it just started out as a Java console program. Clunky, unrefined and not very friendly. Yet, I could see the potential and where Oracle likely wanted to take the product, but wondered as to if they were up to the task. I often thought they would just buy Quest who had an impressive portfolio and was probably (I thought) used most by DBAs, developers, and architects alike. Maybe even Embarcadero, whose DBArtisan tool I actually preferred and found to be a better tool than Quest Toad back then. It is now several years later and how Enterprise Manager has grown up. It is now offered as "DB Console" for single database administration, and as "Grid Control" for multiple database administration and your enterprise-wide monitoring and administration tool.

Grid Control 11g (11.1.0.1) is a web-based tool leveraging WebLogic Server (WLS) 10.3.2 for UI presentation, a database back-end for its storage repository, and agents for data gathering and carrying out monitoring and administration. I'm sure I've been told on many an occasion why each Grid Control will require a specific WLS version but the answer escapes me for now. This however should not be an issue since the WLS should be self-contained (i.e. only for Grid Control) and IMHO installed on the same server as the OMS. If you have multiple OMS for HA, then install a WLS instance on each OMS as well for HA.

Now released is Enterprise Manager 12c (12.1.0.1) Grid Control which will require WLS 10.3.5. This brings a new and improved interface, which also includes cloud computing management features such as charge back and metering.

OEM12.1 - screen shot


My interest in this case are the available upgrade options as shown below:

Two-System Upgrade - Creating a new EM 12.1 infrastructure and switching the new 12.1 agents to point to the new EM 12.1 infrastructure. Requires minimal downtime but new EM infrastructure and you will also have to merge data that was accruing while the upgrade was taking place.

OEM12.1 - 2-system upgrade

One-System Upgrade - This option requires more downtime but no additional infrastructure. Here you are installing the new 12.1 OMS (and 10.3.5 WLS) on the same servers, and then upgrading your OMS to 12.1 (which includes upgrading the repository). Prior to this you would have shutdown your pre-12.1 agents and started up your new 12.1 agents (which were already installed).

OEM12.1 - 1-system upgrade

I prefer the Two-System approach since it has the advantage of minimizing downtime, while providing a fallback option. It also seems a bit less complicated. The previous hardware can be recycled so I don't find that to be an issue (also serves to provide a hardware refresh opportunity). I think most would agree with this being the better of the two options given. To provide a bit more information see the comparison below:

OEM12.1 - upgrade comparison

OEM12.1 - upgrade approaches

OEM12.1 - 2-system upgrade post

OEM12.1 - 1-agent upgrade screen

OEM12.1 - upgrade best practice workflow

Exalytics Business Intelligence Machine Released at Oracle OpenWorld 2011

Yes, this is yet another blog entry about Oracle OpenWorld 2011, and the just announced Oracle Exalytics Business Intelligence Machine. What makes this blog different is that I'm not just re-stating everything that has already been said and announced (at least not entirely), but will provide a bit of compare and contrast.

I've been speculating, along with others, that Oracle was working on some appliance (and of course had to call it 'Exa-something', sigh) that did in-memory computing. Whether this was a response to SAP's HANA in-memory computing solution, or SAP just beat them to the punch can be argued, but that is not the point. The point is that both are here and it's interesting to see how they stack up. Quite frankly, I was expecting TimesTen to be integrated with Exadata to make it more general purpose, but integrating EssBase, BI Suite and creating a separate front-end analytics machine is very welcome and possibly a better solution.

Until recently, via several colleagues attending SAP's annual TechEd conference, I was not able to track down details on HANA. I've managed to learn the certified configurations and more as to what it is meant to accomplish, what is missing (and planned for future), as well as some other helpful information. Even still details are lacking, with SAP preferring you contact a sales agent for full details and an engagement. I much prefer Oracle's policy of providing as much public domain information as possible for all it's products and solutions, as soon as possible. Still several months after introduction, HANA public domain information is limited, while already Oracle has a solution brief available here.


Hardware
Exalytics leverages commodity, off the shelf Oracle/Sun hardware components (model appears to be Sun Fire X4470-M2). The pertinent details are below:


  • CPU: 4 x 10-core Intel E7-4800 (40 processing cores)
  • RAM: 1 TB
  • HDD: direct attached storage (DAS) with 3.6 TB raw capacity (RAID layout)
  • Network: 2 x 40 Gb/s (QDR) Infiniband (IB) ports (for Exadata connection); 2 x 10 GbE ports (for client access); 4 x 1 GbE ports


It comes in single units of 3RU rack-mountable servers which can be clustered for expanded memory and availability. NAS storage is supported for storing shared configuration data and metadata. Based on my understanding, this is meant to connect to Exadata machines (not sure if exclusively) which is where the IB connections come into play. In a multi-Exadata setup, my guess would be that the IB connections are made to a single IB spine switch, similar to how connections to the ZFS appliance are made. This has always been a point of concern as we've suffered an entire environment outage due to a bug in the IB switch taking down the entire IB network, and hence all the Exadata machines. You can read about it here.


Software
Unlike the hardware, the software is not all available outside the Exalytics environment. It consists of:


  • Oracle BI Foundation - provides relational and multi-dimensional OLAP analysis, scorecards, predictive analysis, visualizations, alerts and notifications, search and mobile access. 
  • Oracle Essbase - provides multi-dimensional OLAP cubes with parallel processing enhancements (including others)
  • Oracle TimesTen - provides in-memory caching and SQL access, enhanced for in-memory analytics. Also provides columnar compression support which effectively can turn the 1 TB RAM into 5 TB using Oracle's 5x compression ratio estimate.


The Oracle BI Foundation and Oracle TimesTen have apparently been enhanced to work together and provide features exclusive to the Exalytics platform. Essbase has also been memory optimized. The columnar compression support is interesting from the standpoint of fully supporting Exadata, and turning 1 TB into 5 TB effectively. Clustering is available with multiple units, to not only provide high availability but also to pool the memory resources and accommodate larger data sets. Automated management was mentioned in Larry Ellison's keynote address and is also highlighted in the solution brief document. I've heard this all before, as I'm sure most have, I will reserve judgement until I get a chance to see it in action, but am skeptical when I hear terms such as "intelligent" and "automated". The combination of enhanced versions of TimesTen and EssBase provide:


  • In-memory data replication (TimesTen)  for data sets which can fit at least 2x entirely in memory.
  • In-memory adaptive data mart by EssBase using TimesTen's caching of recent or 'hot' data to update data marts.
  • In-memory intelligent result cache (TimesTen), which I'm pretty sure the intelligent part is a bit of marketing from past experiences.
  • In-memory cubes (EssBase) which are writable.


Administration
Similar to Exadata this is an intergrated, pre-built system which touches several areas of IT, i.e. networking, server, applications (EssBase and BI), and database (TimesTen). Of course, Oracle will prefer, and highly recommend, that companies choose their managed services offering to setup and administer the system. However, this is simply not going to happen in every case and for those who choose to do setup and administration in-house, the same questions will be asked for Exalytics as has been for Exadata and Exalogic. That is, who or what teams will be responsible for the maintenance? Arup Nanda has put forward the idea of an Exadata DBA, but I think this should be taken a bit further. I believe with the introduction of these various appliances (not just Oracle), there needs to be a new job description for "Appliance Administrators" with the specific appliance called out. So there would be an Exalytic Administrator with the necessary skills overlapping each section. If needs be the administrator can always fall back on resources with skill sets for a specific area (such as the networking component).


Closing
I think the Oracle Exalytic BI Machine is a good salvo by Oracle and I look forward to testing it out (hopefully soon) and making public my findings. Let the analytic appliance wars now begin in full force.

Tuesday, July 19, 2011

Oracle 11gR2 (11.2.0.2) Installation - Database software

This is the 2nd in my series of Oracle11gR2 installations, focusing now on the installation of the database or RDBMS software for 11.2.0.2 having already installed the Grid Infrastructure component per my previous post here. If you don't want to install the Grid Infrastructure and use ASM (and Oracle Restart) that is fine, go ahead and skip that first article. In such a case I assume you would be using file systems (perhaps with dNFS which is a post for another time).

Here again, I will be using the command line approach since this is an easy way to have everything scripted and automated (and not require a GUI). I'll show the parameters that need to be adjusted, but if you are not comfortable then I'd suggest doing an initial GUI installation, saving that response file when prompted and then using the saved response file as your gold image for further scripted installations. Note that in the below I use RDBMS_HOME instead of ORACLE_HOME to make the distinction between the actual database home and the grid infrastructure home.

Requirements
See my previous post here

Installation
1. If this is your first installation, then you will want to create the '/etc/oraInst.loc' file, as the root user:

Note: This is optional since you will be prompted at the end of the installation to run $ORACLE_BASE/../oraInventory/orainstRoot.sh which does this for you if this files does not exist.

echo "inventory_loc=/oracle/app/oraInventory
inst_group=dba"
> /etc/oraInst.loc
chown oracle:dba /etc/oraInst.loc
chmod 664 oraInst.loc


2. Edit the DB response file, 'db_inst.rsp' for the values as show below:

ORACLE_HOSTNAME=oradb01
ORACLE_BASE=/oracle/app/oracle
ORACLE_HOME=/oracle/app/product/11.2.0/db_1


3. Run the DB installation (using responsefile and silent installation) as the oracle user:

Note: Ensure you set your DISPLAY environment variable, or you are at run level 5, otherwise you will encounter an error.

./runInstaller -silent -noconfig -responseFile /home/oracle/rsp/db_inst.rsp


4. As the root user run '$RDBMS_HOME/root.sh' when the above completes as prompted.


5. As the oracle user, create an OCM response file. This saves a lot of time later down when you are prompted for those values. Simple run the following and follow the instructions to create and save the response file:

$RDBMS_HOME/OPatch/ocm/bin/emocmrsp


6. Apply the latest OPatch patch (MOS patch ID 6880880), then the latest PSU to this installation as the oracle user. Simple unzip the OPatch patch to the $RDBMS_HOME. For the PSU, unzip to a temporary location, navigate to the patch location and run:

$RDBMS_HOME/OPatch/opatch apply -ocmrf /home/oracle/rsp/ocm.rsp 


7. Apply patch 12431716 (as required by PSU 2) by unzipping to a temporary location, change to the patch directory and as oracle user running: 

$RDBMS_HOME/OPatch/opatch apply -ocmrf /home/oracle/rsp/ocm.rsp


At this point you have fully prepared GI and RDBMS software installations with a ready listener and two disk groups. Now you can create a new database, or migrate an exiting database. I'll leave the new installation to another post in which I'll show how to use DBCA and a template to do it silently, without a GUI.

Tuesday, July 12, 2011

Oracle Exadata Storage Expansion Rack

Today Oracle announced the availability of the Exadata Storage Expansion Rack and I can't say I didn't see this one coming. I don't think my current company was alone in forecasting running low on space within the confines of a single Exadata machine, but not wanting to pay for another full rack of Exadata since the compute nodes were not needed. Sure, you can buy just the additional storage, but it's a pain as from what I've seen that is a negotiation with Oracle again.

There does not seem to be anything different between previously getting just the additional Storage Servers vs. now buying a Storage Expansion Rack. I see this announcement as just Oracle making official or public the fact that you can get the Storage Servers without the compute nodes. As can be expected, the racks are available in the same configurations as the database machine itself, i.e. quarter, half, and full rack, and include IB switches and cables. Since these are the standard configurations, it is a bit curious as to why Oracle choose to nonetheless license the Storage Software separately. Everyone knows the Storage Software costs, and the configuration is fixed so why bother? Sure, you can negotiate discounts for the software costs (as everyone does), but why not just give the full starting price without asking me to go calculate it?

  • Full rack: 18 Oracle Exadata Storage Servers with up to 6.75 TB of smart flash cache, 432 TB of raw disk storage and 216 CPU cores, will run you $750,000 a box.
  • Half rack: 9 Exadata Storage Servers has 3.4 TB of smart flash cache, 216 TB of raw disk storage and 108 CPUs will run you $425,000.
  • Quarter rack: system has 1.5 TB of flash cache, 96 TB of disk storage and 48 CPU cores costs $225,000.

Another point of note is that this is really for backup, storage archival, or low bandwidth processing as only the high capacity or HC disks (7.2K rpm, 2TB) are used in the current expansion racks. Sorry, no high performance (HP) or 15K rpm SAS disks (600GB) yet. Considering this is an expansion rack to augment storage and not necessarily for primary usage this is not to be totally unexpected though it would be nice to have that option also available in case you are in the situation of requiring additional HP storage. Yes, you still get the TB of Smart Flash Cache and the Storage Software but it's still not as fast when you need 15K rpm and true SAS. Maybe this might be a future standard offering as well with enough customer demand.

If you want to read more you can head over to the Oracle Exadata website which has the full details, a direct link to the PDF data sheet is here as well. But happy times are ahead. My prediction is that we will also see some sort of Exadata announcement at OpenWorld 2011 (where's my blogger approval Oracle?) perhaps newer Intel CPUs and faster/more storage and/or a MySQL machine.

Thursday, July 07, 2011

Oracle 11gR2 (11.2.0.2) Installation - Grid Infrastructure

I quite often find myself doing Oracle installations and referencing back to the manuals, read me, other blogs, and various other information sources in an attempt to ensure I get everything just perfect. This is regardless of having done it many times and having documented it myself. These days however, I find this a time consuming exercise, especially given my increasing workload so I thought to simply blog about my method which follows below.

The installation below utilizes a command line only method as I often need to be do multiple installations in a standard, and scripted way. The best method I've found for reaching this point is to initially start with a GUI approach, from which the response file is saved which can then be used for further silent, command line driven installations. I'll start with the installation of the Grid Infrastructure component which is used for ASM and Oracle Restart, and continue with a follow-up posting with the RDBMS software installation. In the method that follows I use a simplified setup with only a single group assigned to the oracle user, i.e. 'dba'. Of course you will need to modify the below for your environment.

Requirements
OS: SUSE Linux Enterprise Service 10.3 (64-bit)

Packages (standard installation):
make-3.80-202.2
binutils-2.16.91.0.5-23.34.33
libstdc++33-3.3.3-7.8.1
gcc-4.1.2_20070115-0.29.6
gcc-c++-4.1.2_20070115-0.29.6
glibc-2.4-31.74.1
glibc-devel-2.4-31.74.1
glibc-devel-32bit-2.4-31.74.1
ksh-93s-59.11.35
libaio-0.3.104-14.2
libaio-32bit-0.3.104-14.2
libaio-devel-0.3.104-14.2
libaio-devel-32bit-0.3.104-14.2
libelf-0.8.5-47.2
libgcc-4.1.2_20070115-0.29.6
libstdc++-4.1.2_20070115-0.29.6
libstdc++-devel-4.1.2_20070115-0.29.6
numactl-0.9.6-3.25.28
sysstat-8.0.4-1.7.27
unixODBC-2.2.12
unixODBC-devel-2.2.12
unixODBC-32bit-2.2.12
oracleasmlib-2.0.4-1.SLE10
oracleasm-2.0.5-7.4.50
oracleasm-support-2.1.4-1.SLE10

Configuration files & settings:
/etc/sysctl.conf
# Oracle settings
kernel.core_uses_pid = 1
kernel.suid_dumpable = 1
kernel.msgmni = 2878
# physical RAM size (8 GB) / pagesize (4096)
kernel.shmall = 2097152
# 1/2 of physical RAM typically, 5 GB here
kernel.shmmax = 5368709120
kernel.shmmni = 4096
#kernel.sem = 250 32000 100 128
kernel.sem = 250 32000 100 142
# 512 x processes (for example 6815744 for 13312 processes)
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.ipv4.conf.default.rp_filter = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
# For RDS use larger value below
#net.core.wmem_max = 2097152
net.core.wmem_max = 1048576
# group ID of dba group
vm.hugetlb_shm_group = 7001
# number of hugepages (2 MB size) to be used for ASMM
vm.nr_hugepages=2560

/etc/security/limits.conf
# Oracle settings
oracle soft core unlimited
oracle hard core unlimited
oracle soft nproc 131072
oracle hard nproc 131072
oracle soft nofile 131072
oracle hard nofile 131072
oracle soft stack 32768
oracle hard stack 32768
oracle soft memlock 5242880
oracle hard memlock 5242880
root soft core unlimited
root hard core unlimited
root soft nproc 131072
root hard nproc 131072
root soft nofile 131072
root hard nofile 131072
root soft memlock 5242880
root hard memlock 5242880

/etc/pam.d/su
# Oracle setting
session  required       pam_limits.so

/etc/pam.d/xdm
# Oracle setting
session  required       pam_limits.so

/etc/sysconfig/ntp
NTPD_OPTIONS="-x -u ntp"    # add '-x' to existing string

~/.ssh/config
Host *
ForwardX11 no    # Set in the oracle user's file

~/.bashrc
if [ -t 0 ]; then
   stty intr ^C
fi    # Disable terminal output commands (assumes bash shell)

Links
ln -s /bin/fuser /sbin/fuser
ln -s /usr/bin/ssh /usr/local/bin/ssh
ln -s /usr/bin/perl /usr/local/bin/perl
ln -s /usr/bin/scp /usr/local/bin/scp



Installation
1. (optional) Create '/etc/oraInst.loc' file as the root user.

Note: This step is optional as it will be done automatically for first time installations when the oraInst.sh script is run.

echo "inventory_loc=/oracle/app/oraInventory
inst_group=dba"
> /etc/oraInst.loc
chown oracle:dba /etc/oraInst.loc
chmod 664 oraInst.loc


2. Edit GI response file, 'grid.rsp' for the below values:

ORACLE_HOSTNAME=oradb01
ORACLE_BASE=/oracle/app/oracle
ORACLE_HOME=/oracle/app/11.2.0/grid_1


3. As the oracle user, run GI installation (using responsefile 'grid.rsp' and silent installation).

Notes: 
   - Ensure your DISPLAY environment variable is set, or you are at run level 5. Otherwise you will not be able to run the command successfully.
   - You may see some warnings for the installation pertaining to a missing library package compat-libstdc++ which can safely be ignored as for SLES 10.3 and greater libstdc++33 is used instead (reference MOS: 'Requirements for Installing Oracle 11gR2 64-bit (AMD64/EM64T) on SLES 10 (Doc ID 884435.1)').

./runInstaller -silent -noconfig -responseFile /home/oracle/rsp/grid.rsp


4. As the root user, run '/oracle/app/oraInventory/oraInst.sh', '$GI_HOME/root.sh' and then '$GI_HOME/crs/install/roothas.pl' (per log file contents) after the above completes. See below for example run of '$GI_HOME/crs/install/roothas.pl':

/oracle/app/11.2.0/grid_1/perl/bin/perl -I/oracle/app/11.2.0/grid_1/perl/lib -I/oracle/app/11.2.0/grid_1/crs/install /oracle/app/11.2.0/grid_1/crs/install/roothas.pl
Using configuration parameter file: /oracle/app/11.2.0/grid_1/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'dba'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node oemgc02 successfully pinned.
Adding daemon to inittab
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
FATAL: Module oracleoks not found.
FATAL: Module oracleadvm not found.
FATAL: Module oracleacfs not found.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9121: Failed to detect control device '/dev/asm/.asm_ctl_spec'.
ACFS-9310: ADVM/ACFS installation failed.
ACFS-9311: not all components were detected after the installation.

oemgc02     2011/06/23 10:02:19     /oracle/app/11.2.0/grid_1/cdata/oemgc02/backup_20110623_100219.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server

4b. To address the failure of the modules as seen above, reference MOS article "ACFS-9327 ACFS-9121 ACFS-9310: ADVM/ACFS installation failed [ID 1265276.1]" and run the following as the root user:

mkdir -p /lib/modules/`uname -r`/extra/usm
cp -p /lib/modules/2.6.16.60-0.54.5*/extra/usm/* /lib/modules/`uname -r`/extra/usm
/sbin/depmod

4c. OHASD will not starting after reboot on SLES (reference MOS: "OHASD not Starting After Reboot on SLES [ID 1325718.1]"). Run the following to fix this issue:

oradb01:~ # chkconfig --list|grep ohasd
init.ohasd                0:off  1:off  2:off  3:off  4:off  5:off  6:off
ohasd                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
oradb01:~ # chkconfig --list|grep raw
raw                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
oradb01:~ # chkconfig raw on
oradb01:~ # chkconfig ohasd on
oradb01:~ # chkconfig --list|grep raw
raw                       0:off  1:off  2:on   3:on   4:off  5:on   6:off
oradb01:~ # chkconfig --list|grep ohasd
init.ohasd                0:off  1:off  2:off  3:off  4:off  5:off  6:off
ohasd                     0:off  1:off  2:off  3:on   4:off  5:on   6:off

Note: Just turning on OHASD will result in error:

oradb01:~ # chkconfig ohasd on
insserv: Service raw has to be enabled for service oracle_has
insserv: exiting now!
/sbin/insserv failed, exit code 1


5. Apply latest OPatch patch, then latest PSU to GI software installation (Oracle Restart Home). As root user run '$GI_HOME/OPatch/opatch auto /oracle/stage/patches_grid -oh $GI_HOME -ocmrf /home/oracle/rsp/ocm.rsp'. An example below:

oemgc02:/oracle/stage/patches_grid # $ORACLE_HOME/OPatch/opatch auto /oracle/stage/patches_grid -oh $ORACLE_HOME -ocmrf /home/oracle/rsp/ocm.rsp
Executing /usr/bin/perl /oracle/app/11.2.0/grid_1/OPatch/crs/patch112.pl -patchdir /oracle/stage -patchn patches_grid -oh /oracle/app/11.2.0/grid_1 -ocmrf /home/oracle/rsp/ocm.rsp -paramfile /oracle/app/11.2.0/grid_1/crs/install/crsconfig_params
opatch auto log file location is /oracle/app/11.2.0/grid_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-06-23_10-32-00.log
Detected Oracle Restart install
Using configuration parameter file: /oracle/app/11.2.0/grid_1/crs/install/crsconfig_params
Successfully unlock /oracle/app/11.2.0/grid_1
patch /oracle/stage/patches_grid/12311357  apply successful for home  /oracle/app/11.2.0/grid_1
patch /oracle/stage/patches_grid/11724916  apply successful for home  /oracle/app/11.2.0/grid_1
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.

5b. Apply patch 12431716, as root user run '$GI_HOME/crs/install/roothas.pl -unlock'. An example below:

oemgc02:/oracle/stage/12431716 # $ORACLE_HOME/crs/install/roothas.pl -unlock
Using configuration parameter file: /oracle/app/11.2.0/grid_1/crs/install/crsconfig_params
Successfully unlock /oracle/app/11.2.0/grid_1

Next, as GI owner run '$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local  -ocmrf /home/oracle/rsp/ocm.rsp'.
Next, as root user run '$GI_HOME/crs/install/roothas.pl -patch'. An example below:

oemgc02:/oracle/stage/12431716 # $ORACLE_HOME/crs/install/roothas.pl -patch
Using configuration parameter file: /oracle/app/11.2.0/grid_1/crs/install/crsconfig_params
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.


6. Create listener from GI home (it will automatically get added to CRS by NETCA). By creating the listener before the ASM or database instances, Clusterware will automatically make the dependency relation. As oracle user edit the netca.rsp file and run:

Note: Ensure you have set the DISPLAY environment variable, or you are at run level 5. Otherwise the command will fail.

$GI_HOME/bin/netca -silent -responsefile /home/oracle/resp/netca.rsp


7. Create an ASM instance in silent mode by using ASMCA. The parameters for ASM in 11.2 require a disk group. I like to create a separate disk group (using a 2 GB disk) for this purpose which also serves the purpose of storing the OCR and voting files should the instance be migrated to RAC (since I also like to store those files using a separate disk group). To create this initial disk group, called SYSTEMDG run ASMCA as the oracle (or GI owner) user, then to create the additional disk groups for DATA and FRA, run similar commands using '-createDiskGroup' instead of '-configureASM'. See the examples below:

oracle@oradb01:~/bin> $GI_HOME/bin/asmca -silent -configureASM -sysAsmPassword sysPassw0rd -asmsnmpPassword asmsnmpPassw0rd -diskGroupName SYSTEMDG -diskList 'ORCL:SYSTEMD' -redundancy EXTERNAL -au_size 4 -compatible.asm '11.2.0.2.0' -compatible.rdbms '11.2.0.2.0'

ASM created and started successfully.

Disk Group SYSTEMDG created successfully.

oracle@oradb01:~/bin> $GI_HOME/bin/asmca -silent -createDiskGroup -sysAsmPassword sysPassw0rd -diskGroupName D1_T2_A4 -diskList 'ORCL:D_T2_*' -redundancy EXTERNAL -au_size 4 -compatible.asm '11.2.0.2.0' -compatible.rdbms '11.2.0.2.0'

Disk Group D1_T2_A4 created successfully.

oracle@oradb01:~/bin> $GI_HOME/bin/asmca -silent -createDiskGroup -sysAsmPassword sysPassw0rd -diskGroupName F1_T2_A4 -diskList 'ORCL:F_T2_*' -redundancy EXTERNAL -au_size 4 -compatible.asm '11.2.0.2.0' -compatible.rdbms '11.2.0.2.0'

Disk Group F1_T2_A4 created successfully.


You have now successfully installed Grid Infrastructure 11.2.0.2, applied a PSU and recommended patch, and created an ASM instance (along with a corresponding listener). At this point you are ready to commence the database installation process which I will include in a follow-up blog.