Learning ASM: Part 2

In the previous post, I installed ASM on a woefully underpowered machine. Things I learned along the way included:

• Don't skip steps. If Oracle says install Grid Infrastructure First, don't start by setting up a user id for the Oracle Database.
• Partitions on a single physical drive will work for experimenting with ASM, but for hopefully obvious reasons is not something you would do on a system that actually supports development or production.
• 11g Release 2 changes a few of the familiar installation steps, especially the Oracle listener.
• It is helpful to plan (and document) your desired disk group layout before you start. ASM will manage everything for you, but that is no substitute for a diagram on a piece of paper to keep everything straight.

LEARNING THE TOOLS: ASMCMD

ASMCMD is the command-line tool for managing ASM. You can also manage it through Enterprise Manager, but that is not installed yet, so not an option. I always like to know how to do things at the command line, since that is always available. For this section I'm using a chapter from the Oracle Database Storage Administrators Guide .

First we need to make sure ORACLE_HOME and ORACLE_SID are set appropriately. This was not a step in the install guide, or I missed it. I added the following to my .bash_profile for oragrid

umask 022 ORACLE_HOME=/u01/app/oragrid/product/11.2.0/grid ORACLE_SID=+ASM export ORACLE_HOME ORACLE_SID PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin export PATH

Now asmcmd will start properly
[oragrid@loclahost ~]$ . .bash_profile [oragrid@loclahost ~]$ asmcmd ASMCMD> ls ASMDATA/ FRA/ FS/

Next I spent some time playing around with ASM commands. shutdown and startup were straightforward and behaved as expected. I exited asmcmd and restarted it with th e-p and -v options to show the current directory in the prompt (-p) and more verbose output (-v). At the asmcmd prompt, 'help' displays the list of valid asm commands. I could go through the manual, but I like to poke around sometimes on my own in this situation.

ls and cd navigate through the directory. ls -l provides a longer list, as I expected.

ASMCMD [+] > cd +ASMDATA/asm/asmparameterfile ASMCMD [+ASMDATA/asm/asmparameterfile] > ls REGISTRY.253.697071913 ASMCMD [+ASMDATA/asm/asmparameterfile] > ls -l Type Redund Striped Time Sys Name ASMPARAMETERFILE MIRROR COARSE SEP 08 23:00:00 Y REGISTRY.253.697071913

Wonder where a file is located?
ASMCMD [+]> find + registry* +ASMDATA/ASM/ASMPARAMETERFILE/REGISTRY.253.697071913

Need to manage the available disk groups?
ASMCMD [+] > ls -l State Type Rebal Name MOUNTED NORMAL N ASMDATA/ MOUNTED NORMAL N FRA/ MOUNTED NORMAL N FS/ ASMCMD [+] > umount FS ASMCMD [+] > ls -l State Type Rebal Name MOUNTED NORMAL N ASMDATA/ MOUNTED NORMAL N FRA/ ASMCMD [+] > mount FS
ASMCMD [+] > ls -l State Type Rebal Name MOUNTED NORMAL N ASMDATA/ MOUNTED NORMAL N FRA/ MOUNTED NORMAL N FS/



At certain points, such as when a new device is added to a disk group, ASM will rebalance the data among the devices in a group to spread the I/O evenly throughout. You can also force a rebalance to happen immediately.

ASMCMD [+] > rebal FRA

While the Rebalance is occuring, ls -l will report the diskgroup with a Y in the Rebal column.

Curios about the I/O load across disks?

ASMCMD [+] > iostat Group_Name Dsk_Name Reads Writes ASMDATA DATA1 7421952 5046272 ASMDATA DATA2 8433664 2883584 ...

Need to check integrity of a disk grou? chkdg
Did your disk develop bad sectors? the remap command will avoid them. (for more info, help remap).

Let's try taking a disk out of a disk group
ASMCMD [+] > offline -G ASMDATA -D DATA3 ORA-15032: not all alterations performed ORA-15283: ASM operation requires compatible.rdbms of 11.1.0.0.0 or higher Really? Why is that?
ASMCMD [+] > lsattr -G ASMDATA -l rdbms
Name Value
compatible.rdbms 10.1.0.0.0
Interesting, the default is to be compatible with a 10g database, not an 11g database. This setting is probably to assist with upgrades, since 10g will be more prevalent still that 11g for a while. We don't have a 10g database right now, so I'll change it to test the offlining. ASMCMD [+] > setattr -G ASMDATA compatible.rdbms 11.1.0.0.0
Now, let's try that offline again:
ASMCMD [+] > offline -G ASMDATA -D DATA3 ASMCMD [+] > lsdsk -G ASMDATA Path ORCL:DATA1 ORCL:DATA2 ORCL:DATA4 ORCL:DATA5 ORCL:DATA6

GETTIN' CRAZY: CREATING AN ACFS VOLUME VIA ASMCMD (FAIL)

To have ASM act as a storage manager, or volume manager, for a general-purpose file system, we first define a volume within a disk group. We have the FS disk group defined for this purpose, so we'll create 2 volumes within that.

ASMCMD [+] > volcreate -G FS -s 2G --redundancy mirror acfs1
ORA-15032: not all alterations performed
ORA-15477: cannot communicate with the volume driver

Apparently something else is not installed. According to documentation, ACFS is only supported on Oracle Enterprise Linux 5 or Red Had Enterprisxe Linux 5. CentOS 5 is supposed to be equivalent to RHEL, perhaps this is a point where it is not equivalent enough?

My lesson to myself here - stick with supported O/S versions. Oracle Enterprise Linux is supported, and designed around running ASM and the Oracle Database. I'll be switching to that later to study ACFS, but in the meantime I'll table the ACFS exploration and continue looking at ASM and 11gR2.

REBOOT: GETTING ASM TO START AUTOMATICALLY ON BOOT

I want ASM to start on boot, so I modified /etc/oratab to start +ASM.

I stopped ASM using ASMCMD
ASMCMD> shutdown --immediate
ASM diskgroups dismounted
ASM instance shutdown
Connected to an idle instance.

The O/S was restarted.

Following the restart, a ps-fu oragrid showed a few processes running, but not enough.
ASMCMD startup yielded an ORA-01078: failure in processing system parameters, and ORA-29701: unable to connect to Cluster Synchronization Service. The supporting cluster software is not fully functioning. To start it:

login as root
cd /u01/app/oragrid/product/11.2.0/grid/bin
./crsctl start resource -all

Now ps -fu oragrid shows an instance running. logout of root, back in as oragrid, and now asmcmd shows the disk groups available. Interestingly, the startup=Y was changed back to N in /etc/oratab for the +ASM instance. Note to self: +ASM should not be managed by oratab.

The installer added oracleasm and ohasd to the /etc/init.d and rc3.d and rc5.d, but not the cluster services. I created an init script to do the steps we just did manually

cd /etc/init.d
cp -p ohasd ocrs

Edit ocrs changing the start() subroutine to include
/u01/app/oragrid/product/11.2.0/grid/bin/crsctl start resource -all
and the stop() subroutine to include
/u01/app/oragrid/product/11.2.0/grid/bin/crsctl stop resource -all
as well as modifying any descriptive text in the script.

I created symbolic links in rc5.d and rc3.d for Start and Stop at step 97 (after OHASD has started)

cd ../rc5.d
ln -s /etc/init.d/ocrs S97ocrs
ln -s /etc/init.d/ocrs K97ocrs

cd ../rc3.d
(same thing)

And now ASM starts on boot.

Next post: actually installing a database on 11gR2 ASM.