---------------------------------------------------------------------------------------
DISCLAIMER:
This post is provided as a log of personal experimentation only. If you attempt to replicate these steps on your system, it is wholly at your own risk with no guarantee you will get the same results as I did. It is in no way supported by Oracle, and I take no responsibility for any ill effects on your system - and Oracle won't either. This is wholly undocumented and unrecommended. I'm purposely changing things to understand them better. If you want a stable (and simpler) experience, use the Linux versions documented to be supported by Oracle.
---------------------------------------------------------------------------------------
With the lawyers held at bay, I can proceed. Remember, don't try this at work, kids - only on a system no one else uses, that you don't mind reinstalling if things go haywire.
My environment (CentOS 5.3)
$ uname -r
2.6.18-128.el5
According to the Oracle Database Storage Administrator's Guide ACFS on a non-clustered system requires some manual steps to launch the volume driver. The steps outlined at the hyperlink above detail what to do manually.
login as root
$ cd /u01/app/oragrid/product/11.2.0/grid/bin
$ ./acfsload start -s
ADVM/ACFS is not supported on centos-release-5-3.el5.centos.1
Not surprising, the documentation said as much. Let's see if we can work around that.
Some backtracking through the shell and perl scripts leads us to ../lib/osds_acfslib.pm
$ cd ../lib
$ cp -p osds_acfslib.pm osds_acfslib.pm.orig
Now I edit osds_acfslib.pm (carefully) Search for "support this release" to get to the section to tweak. (line 278 in my case)
Since CentOS5 is equivalent to RedHat 5, we'll modify the test for RedHat to look like this...
if (($release =~ /enterprise-release-5/) ||
($release =~ /redhat-release-5/) ||
($release =~ /centos-release-5/))
That gets us past the supported version check, based on the version string. Now we get ACFS-9129: ADVM/ACFS not installed
$ mkdir /lib/modules/2.6.18-128.el5/extra/usm
$ cp /u01/app/oragrid/product/11.2.0/grid/install/usm/EL5/i386/2.6.18-8/2.6.18-8.el5-i686/bin/*ko /lib/modules/2.6.18-128.el5/extra/usm
$ chmod 744 /lib/modules/2.6.18-128.el5/extra/usm
This puts the modules in the expected location. We can check it by:
$ cd /u01/app/oragrid/product/11.2.0/grid/bin
$ ./acfsdriverstate -orahome /u01/app/oragrid/product/11.2.0/grid version
ACFS-9205: OS/ADVM,ACFS installed version = 2.6.18-8.el5(i386)/090715.1
Record dependencies for the kernel modules. (The modules won't load if you skip this step).
$ depmod
Now running acfsload start -s completes without any further messages.
In a cluster install the system executables get copied from the install directory to /sbin as part of the root.sh script. Since this was a standalone install, they did not get copied. We get to do that manually (and set permissions).
$ cd /u01/app/oragrid/product/11.2.0/grid/install/usm/EL5/i386/2.6.18-8/2.6.18-8.el5-i686/bin
For each file that does not end in .ko (there are 12 of them):
$ cp acfsdbg /sbin
$ chmod 755 /sbin/acfsdbg
TEST FOR SUCCESS:
Now connect as oragrid.
Running asmca is noticeably different - the Volumes and ASM Cluster File System tabs are available. (They were greyed out when I tried before).
Now we'll try the command that failed before in ASMCMD
ASMCMD > volcreate -G FS -s 2G --redundancy mirror acfs1
This completed without any error message. Switching back to asmca to see it in the GUI...
ACFS1 now appears in the Volume tab, with a device /dev/asm/acfs1-287. It is unmounted. We could also find out this virtual device using the volinfo command in asmcmd.
Now make a file system on that volume and mount it...
$ /sbin/mkfs -t acfs /dev/asm/acfs1-287
mkfs.acfs: version = 11.2.0.1.0.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume = /dev/asm/acfs1-287
mkfs.acfs: volume size = 2147483648
mkfs.acfs: Format complete.
$ /sbin/acfsutil registry -a /dev/asm/acfs1-287 /u01/app/acfsmounts/acfstrialAt this point, the ASM Cluster File Systems tab in asmca shows our registered mount point, in a DISMOUNTED state. ls /u01/app shows no directory .../acfsmounts.
acfsutil registry: mount point /u01/app/acfsmounts/acfstrial successfully added
to Oracle Registry
To mount a filesystem, you need root privilege.
add the following to the beginning of /sbin/mount.acfs, after the header comments
ORACLE_HOME=/u01/app/oragrid/product/11.2.0/grid
Create a directory to mount the filesystem on. This step wasn't in the install guide. I got an error if I didn't pre-create a directory.
$ mkdir /u01/app/acfsmounts/acfstrial
Perform the mount and set permissions
$ mount -t acfs /dev/asm/acfs1-287 /u01/app/acfsmounts/acfstrial
$ chown -R oracle11r2:dba /u01/app/acfsmounts/acfstrial
And we are done! A usable file system, owned by the ID we will later install the Oracle 11gR2 Database using.
$ su - oracle11r2
$ cd /u01/app/acfsmounts/acfstrial
$ ls
lost+found
$ echo hello world > myfile
$ ls
lost+found myfile
$ cat myfile
hello world
And there we have it... ASM Cluster File System running on a standalone CentOS box. Now I have a place to experiment and build up my skill set, with no money out of pocket!
EPILOGUE (TODO):
This doesn't come up automatically following a reboot. I need to add startup scripts to perform the following steps:
1. Load the ADVM drivers
2. Enable the ASM volumes
3. Mount the ACFS filesystems
I can do these steps manually with a combination of the steps above, and with asmca.
Once you've modified osds_acfslib.pm, you can just run acfsroot - it will put everything in the right place for you. Not supported, but nothing you're doing is. :)
ReplyDeleteBrain wondeful piece of research. Thanks for sharing.
ReplyDeleteJust tried this today.
ReplyDelete[root@digest1 /]# /home/grid/app/11.2.0/grid/bin/acfsroot install
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
Make sure you add the install clause to acfsroot script as I shown above otherwise you will get the following error.
[root@digest1 /]# /home/grid/app/11.2.0/grid/bin/acfsroot
acfsroot: ACFS-9134: usage:
acfsroot: ACFS-9337: [arguments]
Hi, It's possible install Oracle9i in ACFS for best performance?
ReplyDeleteThanks.
Alberto
Thanks for sharing this "workaround" I just ran into this error ("ADVM/ACFS is not supported on centos-release-5-5.el5.centos") when running root.sh on each node after installing the 11gR2 Grid Infra. I have a VM environment going with Centos 5.5 64-bit.
ReplyDeleteI'm getting the following error:
ReplyDeleteACFS-9207: You do not have permission to execute Perl in './acfsdriverstate/perl/bin/perl'.
Any help will be greatly appreciated.
Thank you,
vad.roytman@gmail.com
Good work Brian, this helped me to study ACFS on Centos.
ReplyDeleteHi
ReplyDeleteCan we mount acfs volumes by using asmcmd utility or asmca ?