Using udev to configure disks for ASM in Linux

I've never been the biggest fan of Oracle ASMLib. While it is far easier to configure disks for ASM using ASMLib than udev or multipath, it also added an additional requirement for rpms, an additional layer in the stack, and dependency on the kernel version. An internet search will reveal numerious discussions on this very topic, with many having a preference to not use ASMLib. However, due to the increased ease of configuration, strong recommendations from Oracle and believed performance improvements I had always used ASMLib. For what it's worth, there does seem to be some truth behind ASMLib having a performance benefit as though I've not personally tested it, I have read blog postings where its removal resulted in greater CPU usage. If you have enough resources this is not a major concern though over time it may end of being the case.

My stance, and I would think a lot of others, have since changed with the recent developments in Oracle's competition with Red Hat. To be more specific, MOS note "Oracle ASMLib Software Update Policy for Red Hat Enterprise Linux Supportd by Red Hat [ID 1089399.1]", the quote of interest being:

"For RHEL6, Oracle will only provide ASMLib software and updates when configured with a kernel distributed by Oracle. Oracle will not provide ASMLib packages for kernels distributed by Red Hat as part of RHEL6. ASMLib updates will be delivered via Unbreakable Linux Network(ULN) which is available to customers with Oracle Linux support. ULN works with both Oracle Linux or Red Hat Linux installations, but ASMlib usage will require replacing any Red Hat kernel with a kernel provided by Oracle."

With this note, I've since been devoting time to updating my Oracle database standards for Linux, i.e. SLES and RHEL since we are not an Oracle Linux shop and have no plans to pursue such a course (nothing personal Oracle, it's just business). So I began work on what I though was simply making a few modifications to use udev instead of ASMLib following which I'd do due diligence to test it out on our standard versions of SLES and RHEL. I've used it before on RHEL and it should prove simple enough, right?

The Case of the Missing SCSI ID

I was so wrong. When I tried to obtain the SCSI ID for the disks during my testing nothing was returned. I spent some time researching, but came up empty until I realized I was not searching correctly. You see we now run most of our Linux environments on VMware vSphere and I was not using this in my search terms. Correct input and bingo! Seems there is a known issue where by default VMware does not expose SCSI IDs for the disks. To correct this you will need to:

1. Shutdown your VM guest
2. In vCenter, right click your VM guest in the LHS pane and select 'Edit Settings...' (can also be obtained directly via 'Summary' tab)
3. Click the 'Options' tab
4. Select the 'Advanced -> General' navigation item on the left side and click the 'Configuration Parameters...' button displayed on the right
5. Click the 'Add Row' button
6. Add the name 'disk.EnabledUUID' with a value of 'TRUE' (no quotes of any kind) and click 'OK' all the way through the screens to save
7. Restart your VM guest and enjoy receiving SCSI IDs

An alternate method is to use a text editor and add the entry disk.EnabledUUID="TRUE" to your VM guest's VMX file and restart your VM guest.

UDEV Steps
Now onwards with the steps for udev configuration!

1. Add the "options=-g" line to the /etc/scsi_id.config file as the root user.

2. Obtain the SCSI ID for your disks as the root user:

/sbin/scsi_id -g -s /block/{sd_device}

3. Create a udev rules file in /etc/udev/rules.d directory as the root user with entries similar to the below:

vi /etc/udev/rules.d/99-udev-oracle.rules
# ####################################################
# FILE: 99-udev-oracle.rules
# DESC: UDEV rules file for Oracle ASM functionality.
#       Should be placed under /etc/udev/rules.d
# ####################################################
# DATA disks
KERNEL=="sd*",BUS=="scsi",ENV{ID_SERIAL}=="{scsi_id}", NAME="asmdisk1", OWNER="oracle", GROUP="oinstall", MODE="660"
# FRA disks
KERNEL=="sd*",BUS=="scsi",ENV{ID_SERIAL}=="{scsi_id}", NAME="asmdisk2", OWNER="oracle", GROUP="oinstall", MODE="660"

4. Stop and start udev as the root user:

/etc/init.d/boot.udev stop
/etc/init.d/boot.udev start

The SCSI devices can now be accessed by ASM, and you can set your ASM_DISKSTRING parameter to be "/dev/asmdisk*".

Ensure CELL_OFFLOAD_PROCESSING is FALSE in Non-Exadata

A colleague of mine recently brought to my attention a curious wait event that they were experiencing for 'ASM file metadata operation' which occurs when doing operations such as a DROP TABLESPACE, or in this case when using Data Pump. Though the server was basically not doing anything the CPU usage was 100% and there was the strange wait event on 'ASM file metadata operation' with high waits on 'ksv master wait'. So to Oracle Support they went via opening an SR.

It seems that upon applying a PSU to 11.2.0.2 (as of this writing PSU 2 is that latest so likely this will not occur in PSU 3 but that is only my guess) the parameter cell_offload_processing is set to TRUE. In an Exadata environment this is the appropriate setting, however, in a non-Exadata environment, which it was in this case, this causes performance issues to arise as processes on the RDBMS side await on a reply from the ASM side which is trying to delivery smart-scan results.

The quick fix is of course to simply reset the parameter to FALSE, i.e. 'ALTER SYSTEM SET cell_offload_processing = FALSE'. If you prefer you can instead apply patch 11800170. Per the MOS note, "High 'ksv master wait' And 'ASM File Metadata Operation' Waits In Non-Exadata 11g [ID 1308282.1]" this issue is fixed in 11.2.0.3 and 12.1.

Of course 11.2.0.3 is the likely next patch set which is not yet released but is likely in testing. I've been seeing this 12.1 designation on MOS recently which is a good indication that this is the next version for the database, which is likely in early testing. I think that is an entry for a good blog, namely what features I'd like to see in the next database release, and what is likely to be included by Oracle. Attending OpenWorld 2011 would certainly help with that posting, come on Oracle, where is my Blogger approval ;-).

How to use files for ASM disks

One of the issues I've seen is that many people do not have an environment in which they can learn ASM, i.e. they don't have numerous disk to support an ASM setup. For example, if you have a single disk which already has a file system and would like to play around or learn ASM, how do you accomplish this? Turns out this is quite simple in UNIX/Linux using the 'dd' command to create a set of files, which are then associated with loop devices using 'losetup', and associated with raw devices using the 'raw' command.

The steps below are in no way unique so I can not take credit, I also do know the actual originator. I came across the method maybe in 2007 when I was seeking just such a method, and have recently found an article I thought I'd publish in my own words with a bit more detail. I have heard there is a similar method for Windows (makes sense), but as that is not my preferred platform I did not seek to either verify or test out that method.

Note that this setup should not be used in a production environment. It is strictly for testing or training purposes.

1. Create a directory under your file system(s) to store the ASM files.

$> mkdir /u02/asm
$> cd /u02/asm

2. Create files full of zeros using 'dd' command. I've used a block size of 32KB (bs=32K) to improve the build performance, and a count of 983040 to get files of 32GB. The files are named 'asmdiskX' (of=asmdisk1) where X is 1 to 4. I ran the four commands in parallel in the background, which took a really long time (almost 3 hours). No doubt this was due to the fact I was using a shared USB attached 500GB drive (with other things happening) so the operations could have probably gone quicker in serial, with less happening on the drive.

$> dd if=/dev/zero of=asmdisk1 bs=32K count=983040 &
$> dd if=/dev/zero of=asmdisk2 bs=32K count=983040 &
$> dd if=/dev/zero of=asmdisk3 bs=32K count=983040 &
$> dd if=/dev/zero of=asmdisk4 bs=32K count=983040 &

Note that I have run the commands as 'oracle', so I did not need to change ownership. If you have run as root then run the command below on the files to change their ownership to 'oracle':

$> chown oracle:dba /u02/asm/asmdisk*

3. Use 'losetup' to associate loop devices with the regular files (or block devices).

$> losetup /dev/loop1 /u02/asm/asmdisk1
$> losetup /dev/loop2 /u02/asm/asmdisk2
$> losetup /dev/loop3 /u02/asm/asmdisk3
$> losetup /dev/loop4 /u02/asm/asmdisk4

I believe there is a limit on the number of loop devices so you should check before running the commands if you have them available. On my system it was all clear, I've not tested but I believe the command below would create loop devices:

# create a new loop device
$> mknod /dev/loop/300 b 7 300

4. Use the raw command to associate the character block device with a raw device.

$> raw /dev/raw/raw1 /dev/loop1
/dev/raw/raw1: bound to major 7, minor 1

$> raw /dev/raw/raw2 /dev/loop2
/dev/raw/raw2: bound to major 7, minor 2

$> raw /dev/raw/raw3 /dev/loop3
/dev/raw/raw3: bound to major 7, minor 3

$> raw /dev/raw/raw4 /dev/loop4
/dev/raw/raw4: bound to major 7, minor 4

5. Change the ownership of the raw devices to 'oracle' and group to 'dba'.

$> chown oracle.dba /dev/raw/raw[1-4]

6. Setup a startup file to enable the setup to survive a reboot. Place commands from steps 3 to 5 into a script (/etc/init.d/asmsetup) which will run during system startup. An example script and steps are setup is below:

#!/bin/bash
#
# chkconfig: 2345 15 99
# description: Setup files to be used as ASM disks.

# Source function library.
. /etc/init.d/functions

disklocation=/u02/asm

prog=$"ASM file disk setup"

start()
{
echo -n $"Starting $prog: "
/sbin/losetup /dev/loop1 ${disklocation}/asmdisk1
/sbin/losetup /dev/loop2 ${disklocation}/asmdisk2
/sbin/losetup /dev/loop3 ${disklocation}/asmdisk3
/sbin/losetup /dev/loop4 ${disklocation}/asmdisk4
/bin/raw /dev/raw/raw1 /dev/loop1
/bin/raw /dev/raw/raw2 /dev/loop2
/bin/raw /dev/raw/raw3 /dev/loop3
/bin/raw /dev/raw/raw4 /dev/loop4
/bin/chown oracle.dba /dev/raw/raw[1-4]
}

# See how we were called.
case "$1" in
start)
start
;;
*)
echo $"Usage: $0 {start}"
exit 1
esac

$> chkconfig --add asmsetup

This creates the following files:

/etc/rc2.d/S15asmsetup
/etc/rc3.d/S15asmsetup
/etc/rc4.d/S15asmsetup
/etc/rc5.d/S15asmsetup

/etc/rc0.d/K99asmsetup
/etc/rc1.d/K99asmsetup
/etc/rc6.d/K99asmsetup

chkconfig refers the " # chkconfig: 2345 15 99" from asmsetup. This signifies that the service has start run level set to 2, 3, 4 and 5. Stop run level set to 0, 1 and 6 (the stop does nothing in this case). And the start priority should be 15 and stop priority be 99.

ASM now has disks which can be used. I'll write a follow-up on using these disks for an ASM installation. I'll be doing both version 11.1 and 11.2 for ASM but not sure which I'll write about yet (maybe both, maybe one). 11.2 is definately more interesting since I've done 11.1 already so that has the edge right now.