Using udev to configure disks for ASM in Linux

I've never been the biggest fan of Oracle ASMLib. While it is far easier to configure disks for ASM using ASMLib than udev or multipath, it also added an additional requirement for rpms, an additional layer in the stack, and dependency on the kernel version. An internet search will reveal numerious discussions on this very topic, with many having a preference to not use ASMLib. However, due to the increased ease of configuration, strong recommendations from Oracle and believed performance improvements I had always used ASMLib. For what it's worth, there does seem to be some truth behind ASMLib having a performance benefit as though I've not personally tested it, I have read blog postings where its removal resulted in greater CPU usage. If you have enough resources this is not a major concern though over time it may end of being the case.

My stance, and I would think a lot of others, have since changed with the recent developments in Oracle's competition with Red Hat. To be more specific, MOS note "Oracle ASMLib Software Update Policy for Red Hat Enterprise Linux Supportd by Red Hat [ID 1089399.1]", the quote of interest being:

"For RHEL6, Oracle will only provide ASMLib software and updates when configured with a kernel distributed by Oracle. Oracle will not provide ASMLib packages for kernels distributed by Red Hat as part of RHEL6. ASMLib updates will be delivered via Unbreakable Linux Network(ULN) which is available to customers with Oracle Linux support. ULN works with both Oracle Linux or Red Hat Linux installations, but ASMlib usage will require replacing any Red Hat kernel with a kernel provided by Oracle."

With this note, I've since been devoting time to updating my Oracle database standards for Linux, i.e. SLES and RHEL since we are not an Oracle Linux shop and have no plans to pursue such a course (nothing personal Oracle, it's just business). So I began work on what I though was simply making a few modifications to use udev instead of ASMLib following which I'd do due diligence to test it out on our standard versions of SLES and RHEL. I've used it before on RHEL and it should prove simple enough, right?

The Case of the Missing SCSI ID

I was so wrong. When I tried to obtain the SCSI ID for the disks during my testing nothing was returned. I spent some time researching, but came up empty until I realized I was not searching correctly. You see we now run most of our Linux environments on VMware vSphere and I was not using this in my search terms. Correct input and bingo! Seems there is a known issue where by default VMware does not expose SCSI IDs for the disks. To correct this you will need to:

1. Shutdown your VM guest
2. In vCenter, right click your VM guest in the LHS pane and select 'Edit Settings...' (can also be obtained directly via 'Summary' tab)
3. Click the 'Options' tab
4. Select the 'Advanced -> General' navigation item on the left side and click the 'Configuration Parameters...' button displayed on the right
5. Click the 'Add Row' button
6. Add the name 'disk.EnabledUUID' with a value of 'TRUE' (no quotes of any kind) and click 'OK' all the way through the screens to save
7. Restart your VM guest and enjoy receiving SCSI IDs

An alternate method is to use a text editor and add the entry disk.EnabledUUID="TRUE" to your VM guest's VMX file and restart your VM guest.

UDEV Steps
Now onwards with the steps for udev configuration!

1. Add the "options=-g" line to the /etc/scsi_id.config file as the root user.

2. Obtain the SCSI ID for your disks as the root user:

/sbin/scsi_id -g -s /block/{sd_device}

3. Create a udev rules file in /etc/udev/rules.d directory as the root user with entries similar to the below:

vi /etc/udev/rules.d/99-udev-oracle.rules
# ####################################################
# FILE: 99-udev-oracle.rules
# DESC: UDEV rules file for Oracle ASM functionality.
#       Should be placed under /etc/udev/rules.d
# ####################################################
# DATA disks
KERNEL=="sd*",BUS=="scsi",ENV{ID_SERIAL}=="{scsi_id}", NAME="asmdisk1", OWNER="oracle", GROUP="oinstall", MODE="660"
# FRA disks
KERNEL=="sd*",BUS=="scsi",ENV{ID_SERIAL}=="{scsi_id}", NAME="asmdisk2", OWNER="oracle", GROUP="oinstall", MODE="660"

4. Stop and start udev as the root user:

/etc/init.d/boot.udev stop
/etc/init.d/boot.udev start

The SCSI devices can now be accessed by ASM, and you can set your ASM_DISKSTRING parameter to be "/dev/asmdisk*".

Oracle Internet Directory (OID) 11g: Part II - Requirements and Components

The first part of this series on OID 11g focused on purchasing and licensing OID. Now we are ready to focus on what you is needed for a functional OID implementation. Before we get into the details, it is useful to explain what is OID and some of its components.

What is OID?
OID, a component of Fusion Middleware, is a general purpose LDAPv3 compliant directory store. Based on configuration it can scale, be highly available and secure. Most would initially use it for Directory Naming, as a central store for database connectivity details. This would alleviate clients from requiring a local TNSNAMES.ORA file (reduces management and improves efficiency), though instead they would typically require an LDAP.ORA file. This file stores the LDAP connection details, i.e. the OID hostname/IP, connection ports, and realm.

OID can also serve as the central repository for user identification and role information which again reduces management and improves efficiencies. You may think this sounds like Microsoft Active Directory (MSAD) and you would be right! They are both LDAP directories, however, Oracle does not speak with MSAD (or any other LDAP directory) out-of-box directly and this is where OID comes into play. There is also Oracle Virtual Directory (OVD) which acts as a virtual directory service, being a front-end to any LDAP directory and is a valid alternative if you don't have OID or you have multiple directory servers and are okay with extending the directories schema to support the Oracle meta-data. However, you may run into organization constraints extending your MSAD schema which in my case is what occured.

How does OID integrate with MSAD?
There are a few options when integrating OID to your existing LDAP directory store and in this post I'll focus on MSAD though the concepts are similar to other LDAP directory stores with only the details being different. Reviewing the options in the arcticle here (this is rather dated but seems still relevant and its not like Oracle has anything more recent) lead to me to several thoughts. Originally I thought to use the Directory Integration Platform (DIP) tool to synchronize passwords between MSAD and OID but that required installing the Active Directory Password Filter on each domain controller. This of course is simply not practical given the number of AD controllers and corporate policies and was immediately rejected by the Windows Admins. Another option I explored was Server Chaining but that would only work with one AD controller and required a plug-in for password change notification. Even with some creative workarounds such as using a round-robin or LB front end for some of the MSAD controllers, the plug-in had to exist on each domain controller. Again, not practical and rejected.

To take a step back, MSAD and Oracle do not use compatible hashing algorithyms for their passwords so when a user changes their AD password Oracle needs a way to capture that password, and hash it in its own format for transmission to OID. This is why the Password Filter and even the plug-in are required, and on each domain controller since the change may occur on any controller. To be fair we are currently trying to implement an IDM product from another vendor as well and it has run into the same problems so this is not anything specific to OID and MSAD.

At this point it seems my only option is using a combination of Kerberos and Server Chaining. We already use Kerberos via QAS (Quest Authentication Services) for our UNIX/Linux integration to MSAD so their is some experience. Kerberos is rather difficult I've been told and I'll just have to see how things go when I start the detailed implementation in the lab. If anyone has thoughts, experiences or details around how they successfully integrated OID and MSAD please do contact me, I would love to discuss. Such details seem to be not exact in the blogosphere, forums and definitely documentation. My vision is below:

OID Components and requirements
The basic components are:

Database Repository - This database is where OID stores all its information and can be Standard Edition (SE) or Enterprise Edition (EE). Directory Services Plus comes with a restricted use license for SE (as seen here), iAS comes with restricted use of SE and EE (or so it would seem here), though complicating all this is the statement mentiond in my Part I of this series (so I've no idea but am going with what Oracle has stated). You can visit MOS for the current listing of certified versions, I chose 11.2.0.3 EE (single instance). The basic requirements are listed in MOS note 872885.1 and documentation here:

Character Set = AL32UTF8
aq_tm_processes=1
db_block_size=8192
db_cache_size>=144M
dml_locks>=200
java_pool_size>=120M
job_queue_processes>=10
open_cursors>=500 (800 for RAC)
shared_pool_size>=175M
session_max_open_files>=50
sessions>=500
processes>=500 (2500 for RAC)
sga_target>=4G
pga_aggregate_target>=2G
session_cached_cursors>=500
_b_tree_bitmap_plans=FALSE

WebLogic Server (WLS) - I am using OID 11.1.1.5.0 so the certified WLS version is 10.3.5. Note that this is not required if you don't need a nice interface such as ODSM and Fusion Middleware Control to OID. I would recommend though that you install WLS as you lose nothing since you still have command-line and gain a nice interface for those times when you need a pretty face.

Following the installaton of the above required components, you can proceed with the installation of OID itself which, for my own installation consisted of:

Oracle Directory Services Manager (ODSM) - This is a web-based interface for managing instances of OID and OVD and replaces Oracle Directory Manager (ODM) which is now deprecated.

Oracle Enterprise Manager Fusion Middleware Control - This is a graphical interface that provides comprehensive systems management for Oracle Fusion Middleware including OID (target ype). Based on the name you would think you could just use regular OEM but thus far I've not had time to investigate this thoroughly.

Oracle Directory Integration Platform (DIP) - Provides directory synchronization capabilities which includes connectors for out-of-the-box synchronization with Novel eDirectory, and MSAD. This of course is only required if you are seeking such functionality.

Oracle Internet Directory (OID) - The internet directory component itself. Note that I am using version 11.1.1.5.0 which is actually a patchset and requires installation of 11.1.1.2.0 and then the application of the patch ontop. This is a bit strange since there are components within Fusion Middleware 11.1.1.5.0 which are self-contained installations and not patches, much like the Oracle Database 11.2.0.x software. I'm hoping Oracle will give this, along with every Oracle product, similar treatment to be installed as self-contained units instead of patches. I find this leads to clearner, more streamlined and less error-prone installations along with better out-of-place patching.

Note about High Availability
There are many options for HA with OID. The one I've chosen, as shown in my diagram, uses two OID servers each running separate OID and database instances; synchronized via LDAP multi-master replication (MMR), and optional fan-out replicas. Unless you are planning a single OID instance, I do not believe RAC makes a lot of business sense given you get the same features (availability, scalability and performance) along with others (rolling upgrades) from MMR. Using LDAP instead of database MMR provides better granular control over the replication process as an option (security, and performance) though if using SSO you will need to use database MMR. The fan-out replication is essentially just LDAP MMR with the synch process being down stream read only (and possibly filtered).

Oracle Internet Directory (OID) 11g: Part I - Which License?

Well, it has been quite some time since I last posted but I've been kept busy doing a lot of different things at work and of course family life. One such thing that has kept me occupied at work is getting up to speed on Oracle Internet Directory (OID) 11g. My previous experiences with OID were to do with merely using it much like most other people. Yes, I knew how it worked at the high level and of course what it was for, but not the internals such as how to do an installation, configuration, migrations, upgrades, patching, maintenance, backup/recovery, and other fine grained details. To be honest I still don't know a LOT of this stuff as the more I get to know OID the more I realize how much I did not know about directories and their internals and how much I appreciate the need for a separate Identity Management (IDM) Administrator. There is simple a lot to know and do, much like a normal DBA.

Anyways, the point of this series of blogs is to try and help others by exposing a lot of the simple things which I now know. Things such as installation and requirements, backup/recovery, configuration, a few notes on designs and usage, and some license assistance. By far the trickiest thing for me was the licensing which is the focus of this first blog.

How do I get OID?
At first glance this might seem like a simple topic and is exactly what I though. However, OID is bundled, and can be purchased as well, with a variety of software suites such Internet Application Server or iAS, Identity Manager, and Directory Services Plus as examples. You will need to purchase the correct suite, which depends on your actual need or usage, otherwise you may end up buying software and licenses you don't need and spend excessive capital funds as the cost depends on the suite and can be quite a large difference. Case in point:

• Directory Services Plus: $50,000 per processor (+$11,000 for maintenance)
• Internet Application Server (EE): $35,000 per processor (+$7,700 for maintenance)
• Internet Application Server (SEO): $11,500 per processor (+$2,530 for maintenance)
• Internet Application Server (SE): $5,800 per processor (+$1,276 for maintenance)

This listing does not include the other suites in which you can obtain OID, they are just examples as to the varying prices (and options). Of course, each suite also has different pieces, and restricted use licenses for varying included components. The latest pricing information can be obtained here, with FMW11g license information here. I know what some may be thinking that this is just Fusion Middleware. Yes, but Fusion Middleware is the broad software suite name, out of which you purchase individual application suites (such as those I've mentioned). Going through the various options and such is too much for a simple blog posting so I'll just say my current employ licensed OID via iAS as at the time that was the best option. Now it seems the best option, based on our current and future usage, is Directory Services Plus. This means new licenses, or does it?

OID for free!
Apparently, and I say this because regardless of going through 2 months of discussions via email and phone with Oracle sales representatives and product specialists I'm still a little confused, if you are only using OID for Directory Naming, then there is no license to be purchased. Below is the key statements taken from the "Oracle Database License Information 11g Release 2 (11.2)" manual which brought about this conclusion:

"The following restricted-use licenses are included with Oracle Database 11g in the editions indicated:

• A restricted-use license for Oracle Internet Directory (OID) is included with all editions (except for Oracle Database Express Edition) if users use the Directory Naming feature to configure Oracle Net Services. OID may not be used or deployed for other uses. Please contact your Oracle sales representative for additional information on Oracle Internet Directory (OID)."

A colleague of mine was of the opinion this meant either you can license the Oracle DB EE as the repository and use the OID mid-tier piece without cost based on this statement, or license the OID mid-tier and not the repository database (since OID also comes with a restricted use license for using the Oracle database). I though (or more hoped) this meant a license similar to OEM where the OID mid-tier does not need a license (and neither does its restricted use database repository) so long as each remote client database is only using it for Directory Naming. Turns out Oracle was of the same thinking as myself. But the problem is the license seems like such an interpretation. In any case, until we expand usage from Directory Naming into identity management we can save on some capital expenditure (CAPEX) for this financial year.

Conclusion
OID is bundled with many different software suites which all fall under the Fusion Middleware umbrella. Choosing the correct suite is important in getting the best deal, but if you only need the Directory Naming functionality it is already included in your database license purchase (so there is no cost). I would strongly recommend you check this out with your own Oracle sales representatives and if you are given a different response please do let me know! It has been my experience that licenses are different based on who you ask within Oracle and what type of relationship you have with Oracle (i.e. how big or important of a customer you are). You may disagree with this statement but that has been my experience.

In my next piece I will explore the requirements and installation of some required components for OID11g.