It tracks the OS resource consumption at each node, process, and device level continuously. It collects and analyzes the cluster-wide data. In real time mode, when thresholds are hit, an alert is shown (i.e. you need to be looking at the screen in GUI mode). For root cause analysis, historical data can be replayed to understand what was happening at the time of failure.
I had a bit of trouble during the initial installation since it was not made clear in the README that you MUST unzip and run the subsequent installation process from the home directory of the user doing the installation. I forgot this during re-installation as well, which led to me spend quite a lot of time re-learning this important fact, and hence documenting this for reference.
Oracle Enterprise Manager (OEM) Grid Control, and other tools, provide similar functionality, but I've found that Cluster Health Monitor (CHM) is better in terms of real-time analysis, monitoring and its playback functionality. It is also free, as in no charge, and very easy to setup and maintain.
My environment:
- 4-node Oracle 11.2.0.1 RAC
- Hostnames: ucstst11, ucstst12, ucstst13, ucstst14
- OS: Oracle Enterprise Linux 5u4 x64
Installation Steps
1. Create 'crfuser' on all nodes in cluster - This can be any user except root, I do recommend however using a separate user from the regular 'oracle' for separation of duty. Perhaps a standard user used for monitoring tools. The user's group does not have to be oinstall, I just use it here to show that this user also owns 'Oracle' software, and will be an administrator.
#> useradd -d /opt/crfuser -m -s /bin/ksh -g oinstall crfuser
2. Setup password-less SSH user equivalence among all the users across the nodes - I wont go into this here since there are numerous sites as to how to set this up (including one of my own I believe).
Note: If you do have a previous installation of CHM installed you will need to uninstall before proceeding with the new installation. To install, on each node where there is a previous installation:
a. Disable CHM
#> /etc/init.d/init.crfd disable
b. Uninstall CHM
#> /usr/lib/oracrf/install/crfinst.pl -d
c. Remove the Berkely database or BDB (wherever it was installed), and the previus CHM installation home if it still exists.
3. Login as the CHM user ('crfuser') and unzip the install file, it can be obtained from Oracle Technology Network (OTN) here.
#> su - crfuser
#> unzip crfpack-linux.zip
Ensure you are in the 'crfuser' home directory when running the unzip command, or that the unzip is done to that directory.
4. Run the installer to setup the nodes - This step does not actually do any installation so if anything fails simply remove the files, fix the error, and try again. What this step does is to:
a. Run some checks for SSH access to all nodes
b. Assign a replica node
c. Generate a cluster-wide configuration file
d. Send the installation files across the other nodes.
To start the process, from any node:
$> cd ~/install
$> ./crfinst.pl -i ucstst11,ucstst12,ucstst13,ucstst14 -b /opt/oracrfdb -m ucstst11 -N RACDB
Replace the node list with those of your own, you will also need to designate one as a master node, i.e. '-m ucstst11'. If you leave out this argument you will be prompted to designate a master node. By default the cluster will be called 'MyCluster', I've used '-N RACDB' to name my cluster something more known to me, i.e. 'RACDB'. I've also specified (using '-b /opt/oracrfdb') that the location for the BDB database which stores all the captured data should be '/opt/oracrfdb'. This must be a separate file system or mount point, i.e. not under '/' directly otherwise the installation will complain and exit. I'm unsure as to where the default would be located or if there is one since I never tested this setup, but of course you can also work around using somewhere under '/' (if you are short on mounts or space) by modifying the appropriate line in the 'crfinst.pl' Perl script, or using a loopback filesytem.
5. Following this, you will then need to run on each node as the root user:
#> /opt/crfuser/install/crfinst.pl -f -b /opt/oracrfdb -N RACDB
#> /etc/init.d/init.crfd enable
Note: The above step will remove the files which were unzipped previously in the 'crfuser' home directory from the 'crfpack-linux.zip' file. So don't be surprised when this happens.
6. Install the GUI - It is not recommended to install the GUI on any of the RAC nodes due to performance (the GUI takes a few resources) and availability issues (the node running the GUI goes down), but in a simple setup this is okay. Ideally you would install only the GUI portion on any remote client machine. To install only the GUI:
a. Unzip the 'crfpack-linux.zip' file again, to the home directory of the 'crfuser' (or whatever user this time around).
$> cd /opt/crfuser
$> unzip crfpack-linux.zip
b. Run the GUI installation
$> mkdir oracrfgui
$> cd install
$> ./crfinst.pl -g /opt/crfuser/oracrfgui
7. Run the GUI. The below command starts the GUI by connecting to the master node, with a one second refresh time.
$> cd ~/oracrf/bin
$> ./crfgui -m ucstst11
Some Usage Tips
Refresh Rate
To modify the GUI refresh rate (default is 1 second) specify the '-r
$> ./crfgui -r 5 -m ucstst11
Historical Mode
To do historical analysis specify the '-d
Creating a Loopback Filesystem for the BDB location
1. Create the local file on each node which must be at least 5GB per node, i.e. 5000 * N, or in my case 20000 (5000 * 4).
#> dd if=/dev/zero of=/mnt/oracrfdb.fs count=20000 bs=1M
2. Create a file system on the file, using a label of "ipdosbdb' or whatever is your choice:
#> mke2fs -F -j -L "ipdosbdb" /mnt/oracrfdb.fs
3. Edit the '/etc/fstab' file to ensure the filesystem will be re-mounted following a reboot by adding the following line:
Specify an available loopback device substituting '/dev/loop0' as appropriate.
4. Mount the file system:
#> mount /opt/oracrfdb
Hope this helps. Feedback and comments welcome!
