Friday, September 18, 2009

Learning ASM: Breakin' the Rules (or, Running ACFS on CentOS)

Part 2 mentioned that I could not get ACFS to work on CentOS. This is, of course, because it is not supported. I did find a way to work around this.

---------------------------------------------------------------------------------------
DISCLAIMER:

This post is provided as a log of personal experimentation only. If you attempt to replicate these steps on your system, it is wholly at your own risk with no guarantee you will get the same results as I did. It is in no way supported by Oracle, and I take no responsibility for any ill effects on your system - and Oracle won't either. This is wholly undocumented and unrecommended. I'm purposely changing things to understand them better. If you want a stable (and simpler) experience, use the Linux versions documented to be supported by Oracle.

---------------------------------------------------------------------------------------

With the lawyers held at bay, I can proceed. Remember, don't try this at work, kids - only on a system no one else uses, that you don't mind reinstalling if things go haywire.

My environment (CentOS 5.3)

$ uname -r
2.6.18-128.el5

According to the Oracle Database Storage Administrator's Guide ACFS on a non-clustered system requires some manual steps to launch the volume driver. The steps outlined at the hyperlink above detail what to do manually.

login as root
$ cd /u01/app/oragrid/product/11.2.0/grid/bin
$ ./acfsload start -s
ADVM/ACFS is not supported on centos-release-5-3.el5.centos.1

Not surprising, the documentation said as much. Let's see if we can work around that.

Some backtracking through the shell and perl scripts leads us to ../lib/osds_acfslib.pm
$ cd ../lib
$ cp -p osds_acfslib.pm osds_acfslib.pm.orig

Now I edit osds_acfslib.pm (carefully) Search for "support this release" to get to the section to tweak. (line 278 in my case)

Since CentOS5 is equivalent to RedHat 5, we'll modify the test for RedHat to look like this...

if (($release =~ /enterprise-release-5/) ||
($release =~ /redhat-release-5/) ||
($release =~ /centos-release-5/))

That gets us past the supported version check, based on the version string. Now we get ACFS-9129: ADVM/ACFS not installed

$ mkdir /lib/modules/2.6.18-128.el5/extra/usm
$ cp /u01/app/oragrid/product/11.2.0/grid/install/usm/EL5/i386/2.6.18-8/2.6.18-8.el5-i686/bin/*ko /lib/modules/2.6.18-128.el5/extra/usm
$ chmod 744
/lib/modules/2.6.18-128.el5/extra/usm

This puts the modules in the expected location. We can check it by:
$ cd /u01/app/oragrid/product/11.2.0/grid/bin
$ ./acfsdriverstate -orahome
/u01/app/oragrid/product/11.2.0/grid version
ACFS-9205: OS/ADVM,ACFS installed version = 2.6.18-8.el5(i386)/090715.1

Record dependencies for the kernel modules. (The modules won't load if you skip this step).

$ depmod

Now running acfsload start -s completes without any further messages.


In a cluster install the system executables get copied from the install directory to /sbin as part of the root.sh script. Since this was a standalone install, they did not get copied. We get to do that manually (and set permissions).

$ cd /u01/app/oragrid/product/11.2.0/grid/install/usm/EL5/i386/2.6.18-8/2.6.18-8.el5-i686/bin

For each file that does not end in .ko (there are 12 of them):
$ cp acfsdbg /sbin
$ chmod 755 /sbin/acfsdbg


TEST FOR SUCCESS:

Now connect as oragrid.
Running asmca is noticeably different - the Volumes and ASM Cluster File System tabs are available. (They were greyed out when I tried before).

Now we'll try the command that failed before in ASMCMD

ASMCMD > volcreate -G FS -s 2G --redundancy mirror acfs1

This completed without any error message. Switching back to asmca to see it in the GUI...
ACFS1 now appears in the Volume tab, with a device /dev/asm/acfs1-287. It is unmounted. We could also find out this virtual device using the volinfo command in asmcmd.


Now make a file system on that volume and mount it...


$ /sbin/mkfs -t acfs /dev/asm/acfs1-287

mkfs.acfs: version = 11.2.0.1.0.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume = /dev/asm/acfs1-287
mkfs.acfs: volume size = 2147483648
mkfs.acfs: Format complete.

$ /sbin/acfsutil registry -a /dev/asm/acfs1-287 /u01/app/acfsmounts/acfstrial

acfsutil registry: mount point /u01/app/acfsmounts/acfstrial successfully added
to Oracle Registry

At this point, the ASM Cluster File Systems tab in asmca shows our registered mount point, in a DISMOUNTED state. ls /u01/app shows no directory .../acfsmounts.

To mount a filesystem, you need root privilege.

add the following to the beginning of /sbin/mount.acfs, after the header comments

ORACLE_HOME=/u01/app/oragrid/product/11.2.0/grid

Create a directory to mount the filesystem on. This step wasn't in the install guide. I got an error if I didn't pre-create a directory.

$ mkdir /u01/app/acfsmounts/acfstrial

Perform the mount and set permissions

$ mount -t acfs /dev/asm/acfs1-287 /u01/app/acfsmounts/acfstrial
$ chown -R oracle11r2:dba /u01/app/acfsmounts/acfstrial

And we are done! A usable file system, owned by the ID we will later install the Oracle 11gR2 Database using.

$ su - oracle11r2

$ cd /u01/app/acfsmounts/acfstrial
$ ls
lost+found
$ echo hello world > myfile
$ ls
lost+found myfile
$ cat myfile
hello world


And there we have it... ASM Cluster File System running on a standalone CentOS box. Now I have a place to experiment and build up my skill set, with no money out of pocket!


EPILOGUE (TODO):

This doesn't come up automatically following a reboot. I need to add startup scripts to perform the following steps:

1. Load the ADVM drivers
2. Enable the ASM volumes
3. Mount the ACFS filesystems

I can do these steps manually with a combination of the steps above, and with asmca.

8 comments:

  1. Once you've modified osds_acfslib.pm, you can just run acfsroot - it will put everything in the right place for you. Not supported, but nothing you're doing is. :)

    ReplyDelete
  2. Brain wondeful piece of research. Thanks for sharing.

    ReplyDelete
  3. Just tried this today.


    [root@digest1 /]# /home/grid/app/11.2.0/grid/bin/acfsroot install
    ACFS-9300: ADVM/ACFS distribution files found.
    ACFS-9307: Installing requested ADVM/ACFS software.
    ACFS-9308: Loading installed ADVM/ACFS drivers.
    ACFS-9321: Creating udev for ADVM/ACFS.
    ACFS-9323: Creating module dependencies - this may take some time.
    ACFS-9327: Verifying ADVM/ACFS devices.
    ACFS-9309: ADVM/ACFS installation correctness verified.

    Make sure you add the install clause to acfsroot script as I shown above otherwise you will get the following error.

    [root@digest1 /]# /home/grid/app/11.2.0/grid/bin/acfsroot
    acfsroot: ACFS-9134: usage:
    acfsroot: ACFS-9337: [arguments]

    ReplyDelete
  4. Hi, It's possible install Oracle9i in ACFS for best performance?

    Thanks.
    Alberto

    ReplyDelete
  5. Thanks for sharing this "workaround" I just ran into this error ("ADVM/ACFS is not supported on centos-release-5-5.el5.centos") when running root.sh on each node after installing the 11gR2 Grid Infra. I have a VM environment going with Centos 5.5 64-bit.

    ReplyDelete
  6. I'm getting the following error:
    ACFS-9207: You do not have permission to execute Perl in './acfsdriverstate/perl/bin/perl'.
    Any help will be greatly appreciated.
    Thank you,
    vad.roytman@gmail.com

    ReplyDelete
  7. Good work Brian, this helped me to study ACFS on Centos.

    ReplyDelete
  8. Hi
    Can we mount acfs volumes by using asmcmd utility or asmca ?

    ReplyDelete