Adding additional disk to an ASM Disk Group on a Linux host

I wanted to increase the size of the DATA ASM Disk Group, so I created some additional volumes vol05 on the Storwize V7000 and mapped them onto my RedHat Linux host.
The UID of the new volume vol05 that is created is 6005076802828000C000000000000054

I now want the Linux host to see the volume without doing a reboot of the system. To do that I follow the steps mentioned below.

To find the HBA Host ids for the system, do the following as root on the Linux system.

[root@isvx7 ~]#  ls /sys/class/fc_host host5  host6

Scan the SCSI bus to detect the new volume. Run the below command on the Linux host that is attached to the storage system. There must be a space between each of the minus signs contained within the double quotes

[root@isvx7 ~]#  echo "- - -" > /sys/class/scsi_host/host5/scan [root@isvx7 ~]#  echo "- - -" > /sys/class/scsi_host/host6/scan

[root@isvx7 ~]# multipath -d -l mulipath.conf line 111, invalid keyword: prio mpath145 (36005076802828000c000000000000054) dm-4 IBM,2145 [size=300G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active]  \_ 5:0:0:4 sdah 66:16  [active][undef]  \_ 5:0:1:4 sdai 66:32  [active][undef]  \_ 5:0:2:4 sdaj 66:48  [active][undef]  \_ 5:0:3:4 sdak 66:64  [active][undef]  \_ 6:0:0:4 sdal 66:80  [active][undef]  \_ 6:0:1:4 sdam 66:96  [active][undef]  \_ 6:0:2:4 sdan 66:112 [active][undef]  \_ 6:0:3:4 sdao 66:128 [active][undef] mpath144 (36005076802828000c000000000000050) dm-3 IBM,2145 [size=300G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=0][active]  \_ 5:0:3:3 sdaf 65:240 [active][undef]  \_ 6:0:3:3 sdag 66:0   [active][undef]  \_ 5:0:0:3 sdh  8:112  [active][undef]  \_ 6:0:0:3 sdi  8:128  [active][undef]  \_ 6:0:1:3 sdp  8:240  [active][undef]  \_ 5:0:1:3 sdq  65:0   [active][undef]  \_ 5:0:2:3 sdx  65:112 [active][undef]  \_ 6:0:2:3 sdy  65:128 [active][undef] mpath143 (36005076802828000c000000000000053) dm-2 IBM,2145 [size=300G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=0][active]  \_ 5:0:3:2 sdad 65:208 [active][undef]  \_ 6:0:3:2 sdae 65:224 [active][undef]  \_ 5:0:0:2 sdf  8:80   [active][undef]  \_ 6:0:0:2 sdg  8:96   [active][undef]  \_ 5:0:1:2 sdn  8:208  [active][undef]  \_ 6:0:1:2 sdo  8:224  [active][undef]  \_ 5:0:2:2 sdv  65:80  [active][undef]  \_ 6:0:2:2 sdw  65:96  [active][undef] mpath142 (36005076802828000c000000000000052) dm-1 IBM,2145 [size=300G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=0][active]  \_ 5:0:3:1 sdab 65:176 [active][undef]  \_ 6:0:3:1 sdac 65:192 [active][undef]  \_ 5:0:0:1 sdc  8:32   [active][undef]  \_ 6:0:0:1 sde  8:64   [active][undef]  \_ 6:0:1:1 sdl  8:176  [active][undef]  \_ 5:0:1:1 sdm  8:192  [active][undef]  \_ 5:0:2:1 sdt  65:48  [active][undef]  \_ 6:0:2:1 sdu  65:64  [active][undef] mpath141 (36005076802828000c000000000000051) dm-0 IBM,2145 [size=300G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=0][active]  \_ 5:0:3:0 sdaa 65:160 [active][undef]  \_ 5:0:0:0 sdb  8:16   [active][undef]  \_ 6:0:0:0 sdd  8:48   [active][undef]  \_ 6:0:1:0 sdj  8:144  [active][undef]  \_ 5:0:1:0 sdk  8:160  [active][undef]  \_ 6:0:2:0 sdr  65:16  [active][undef]  \_ 5:0:2:0 sds  65:32  [active][undef]  \_ 6:0:3:0 sdz  65:144 [active][undef] [root@isvx7 ~]#

The volumes  mapath141, mpath142, mpath143, and mpath144 are already part of the ASM diskgroup +DATA.

[root@isvx7 ~]# ls -l /dev/mapper/ total 0 crw------- 1 root   root  10, 63 Sep 28 09:57 control brw-rw---- 1 oracle dba  253,  0 Sep 29 14:17 mpath141 brw-rw---- 1 oracle dba  253,  1 Sep 29 14:14 mpath142 brw-rw---- 1 oracle dba  253,  2 Sep 29 14:17 mpath143 brw-rw---- 1 oracle dba  253,  3 Sep 29 14:16 mpath144 brw-rw---- 1 root   disk 253,  4 Sep 29 14:07 mpath145

Change the owner and group of mpath145 to oracle:dba

[root@isvx7 mapper]# chown -R oracle:dba mpath145 [root@isvx7 mapper]# ls -l /dev/mapper/ total 0 crw------- 1 root   root  10, 63 Sep 28 09:57 control brw-rw---- 1 oracle dba  253,  0 Sep 29 14:19 mpath141 brw-rw---- 1 oracle dba  253,  1 Sep 29 14:19 mpath142 brw-rw---- 1 oracle dba  253,  2 Sep 29 14:19 mpath143 brw-rw---- 1 oracle dba  253,  3 Sep 29 14:19 mpath144 brw-rw---- 1 oracle dba  253,  4 Sep 29 14:07 mpath145

The shows the initial distribution of data between the volumes of the diskgroup.

SQL> select name, path, free_mb, total_mb from v$asm_disk; NAME ------------------------------ PATH --------------------------------------------------------------------------------    FREE_MB   TOTAL_MB ---------- ---------- /dev/mapper/mpath145          0          0 DATA_0003 /dev/mapper/mpath144     155443     307200 NAME ------------------------------ PATH --------------------------------------------------------------------------------    FREE_MB   TOTAL_MB ---------- ---------- DATA_0002 /dev/mapper/mpath143     155437     307200 DATA_0001 /dev/mapper/mpath142 NAME ------------------------------ PATH --------------------------------------------------------------------------------    FREE_MB   TOTAL_MB ---------- ----------     155434     307200 DATA_0000 /dev/mapper/mpath141     155438     307200 SQL>

Then using "asmca" I added the additional Storwize V7000 volumes to the DATA ASM Disk Group

After adding the v$asm_disk shows that the data in the volumes has been redistributed between the volumes.
As shown below we can see that /dev/mapper/mpath45 now has data on it. This took roughly 20 to 30 minutes.

SQL>  select name, path, free_mb, total_mb from v$asm_disk; NAME ------------------------------ PATH --------------------------------------------------------------------------------    FREE_MB   TOTAL_MB ---------- ---------- DATA_0003 /dev/mapper/mpath144     185789     307200 DATA_0002 /dev/mapper/mpath143     185789     307200 NAME ------------------------------ PATH --------------------------------------------------------------------------------    FREE_MB   TOTAL_MB ---------- ---------- DATA_0001 /dev/mapper/mpath142     185789     307200 DATA_0000 /dev/mapper/mpath141 NAME ------------------------------ PATH --------------------------------------------------------------------------------    FREE_MB   TOTAL_MB ---------- ----------     185789     307200 DATA_0004 /dev/mapper/mpath145     185792     307200 SQL>

Creating ASM Disk Group with the remaining volumes of the Storwize V7000

In the last post I had used 4 of the volumes to create an ASM Disk Group called DATA. In this post I'm going to use the remaining 2 volumes to create another ASM Disk Group call LOG.

First I'll change the ownership of the volumes to oracle, and group to dba:
[root@isvx3 ~]# chown -R oracle:dba /dev/mapper/mpath4
[root@isvx3 ~]# chown -R oracle:dba /dev/mapper/mpath5

I'll use asmca, a utility that is part of the Oracle 11g R2 grid infrastructure to create the ASM Disk Group
-bash-3.2$ echo $GRID_HOME
/u02/app/oracle/product/11.2.0/grid
-bash-3.2$ $GRID_HOME/bin/asmca

I'm now going to quickly verify if the ASM Disk Group was created properly:





-bash-3.2$ $ORACLE_HOME/bin/asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576   2048000  2047803               50         1023876              0             N  DATA/
MOUNTED  NORMAL  N         512   4096  1048576   1024000  1023886                0          511943              0             N  LOG/

ATTENTION:
Just when everything looked fine, I thought that I should try a reboot and see how things hold up.
After the reboot, the Linux host came up and I ran the "asmcmd lsdg" and this is what I observer.
-bash-3.2$ $ORACLE_HOME/bin/asmcmd lsdg
-bash-3.2$

Yes, nulla, zot, nothing is what I saw.

-bash-3.2$ $ORACLE_HOME/bin/asmcmd lsdg
-bash-3.2$

In the Oracle docs "Oracle® Database Installation Guide11g Release 2 (11.2) for Linux E16763-06 " it says
"The O_DIRECT parameter enables direct read and writes to block devices, avoiding
kernel overhead. With Oracle Database Release 10.2 and later, Oracle Database files are
configured by default to use direct input/output.
With the 2. 6 kernel or later for Red Hat Enterprise Linux, Oracle Linux, and SUSE
Enterprise Server, you must create a permissions file to maintain permissions on
Oracle database files. If you do not create this permissions file, then permissions on
disk devices revert to their default values, root:disk, and Oracle database fails to
start."

I quick and easy way to workaround this issue is to add the following to the /etc/rc.local file
chown oracle:dba /dev/mapper/mpath*
chmod 600 /dev/mapper/mpath*

A more permanent fix would be to add the permission file as shown in the link below:
http://download.oracle.com/docs/cd/B28359_01/install.111/b32002/pre_install.htm#CDEBHDCD

Mapping Storage Array(SAN) Volumes to RedHat 5.5 Linux host

I created 6 volumes on my Storwize V7000 Storage Array, and mapped them to my RedHat Linux 5.5 host. Each volume that I created has a unique UID assigned to it eg. vol01 has UID 60050768018103ABF000000000000001, while vol02 has UID 60050768018103ABF000000000000002 and so on.

Find the HBA host ids of the Linux system.

[root@isvx3 ~]# ls /sys/class/fc_host
host5  host6

Next scan the SCSI but to detect the new volumes.

[root@isvx3 ~]# echo "- - -" > /sys/class/scsi_host/host5/scan
[root@isvx3 ~]# echo "- - -" > /sys/class/scsi_host/host6/scan

On my linux host side I enabled the native multipathing(Device Mapper-Multipath). Now on doing "multipath -d -l" from the command line I see the following:

[root@isvx3 vdbench]# multipath -d -l
mpath1 (3600507680192828b8800000000000002) dm-3 IBM,2145
[size=250G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 6:0:4:0 sdad 65:208 [active][undef]
 \_ 6:0:5:0 sdae 65:224 [active][undef]
 \_ 6:0:6:0 sdaf 65:240 [active][undef]
 \_ 6:0:7:0 sdag 66:0   [active][undef]
 \_ 5:0:4:0 sdn  8:208  [active][undef]
 \_ 5:0:5:0 sdo  8:224  [active][undef]
 \_ 5:0:6:0 sdp  8:240  [active][undef]
 \_ 5:0:7:0 sdq  65:0   [active][undef]
mpath9 (360050768018103abf000000000000006) dm-2 IBM,2145
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 6:0:1:5 sdac 65:192 [active][undef]
 \_ 5:0:0:5 sdg  8:96   [active][undef]
 \_ 5:0:1:5 sdm  8:192  [active][undef]
 \_ 6:0:0:5 sdw  65:96  [active][undef]
mpath8 (360050768018103abf000000000000001) dm-1 IBM,2145
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 6:0:1:4 sdab 65:176 [active][undef]
 \_ 5:0:0:4 sdf  8:80   [active][undef]
 \_ 5:0:1:4 sdl  8:176  [active][undef]
 \_ 6:0:0:4 sdv  65:80  [active][undef]
mpath7 (360050768018103abf000000000000005) dm-0 IBM,2145
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 6:0:1:3 sdaa 65:160 [active][undef]
 \_ 5:0:0:3 sde  8:64   [active][undef]
 \_ 5:0:1:3 sdk  8:160  [active][undef]
 \_ 6:0:0:3 sdu  65:64  [active][undef]
mpath6 (360050768018103abf000000000000004) dm-6 IBM,2145
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 5:0:0:2 sdd  8:48   [active][undef]
 \_ 5:0:1:2 sdj  8:144  [active][undef]
 \_ 6:0:0:2 sdt  65:48  [active][undef]
 \_ 6:0:1:2 sdz  65:144 [active][undef]
mpath5 (360050768018103abf000000000000003) dm-5 IBM,2145
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 5:0:0:1 sdc  8:32   [active][undef]
 \_ 5:0:1:1 sdi  8:128  [active][undef]
 \_ 6:0:0:1 sds  65:32  [active][undef]
 \_ 6:0:1:1 sdy  65:128 [active][undef]
mpath4 (360050768018103abf000000000000002) dm-4 IBM,2145
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 5:0:0:0 sdb  8:16   [active][undef]
 \_ 5:0:1:0 sdh  8:112  [active][undef]
 \_ 6:0:0:0 sdr  65:16  [active][undef]
 \_ 6:0:1:0 sdx  65:112 [active][undef]
[root@isvx3 vdbench]#

From this we can see that volume vol01 maps to dm-1, vol02 to dm-4, vol03 to dm-5, vol04 to dm-6, vol05 to dm-0, and vol06 to dm-2

After changing the owner and group of the volumes to oracle and dba:

[root@isvx3 ~]# chown -R oracle:dba /dev/mapper/mpath6
[root@isvx3 ~]# chown -R oracle:dba /dev/mapper/mpath7
[root@isvx3 ~]# chown -R oracle:dba /dev/mapper/mpath8
[root@isvx3 ~]# chown -R oracle:dba /dev/mapper/mpath9

I then used 4 of these volumes to create an Oracle ASM Disk Group during my 11g R2 Grid Infrastructure installation.

Upgrading from Java version 1.4.2 to 1.6.0 on RedHat Linux

I was planning on running vdbench to get some performance numbers for the IBM Storwize V7000. I installed vdbench, and then decided to bring up it's GUI with the ./vdbench -gui command, that is when it told me that the Java version on my RedHat Linux machine was1.4 , and that I need to update it to 1.5 or a later version.

I downloaded Java from http://www.java.com/en/download/help/5000011400.xml  and installed 1.6 under /usr/java

The current java on my RedHat host machine was linked to:
[root@isvx3 bin]# ls -l /usr/bin/javalrwxrwxrwx 1 root root 22 Nov 22 12:42 /usr/bin/java -> /etc/alternatives/java
[root@isvx3 java]#java -version
java version "1.4.2"
gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-48)

Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
[root@isvx3 java]#




I removed that link:
[root@isvx3 bin]# rm /usr/bin/java
rm: remove symbolic link `/usr/bin/java'? y


Re-linked the jre1.6 which I had installed under /usr/java:
[root@isvx3 bin]# ln -s /usr/java/jre1.6.0_22/bin/java /usr/bin/java
[root@isvx3 bin]# ls -l /usr/bin/java
lrwxrwxrwx 1 root root 30 Nov 22 13:01 /usr/bin/java -> /usr/java/jre1.6.0_22/bin/java




Check to see if the Java version has been updated:
[root@isvx3 bin]# which java
/usr/bin/java
[root@isvx3 bin]# java -version
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
[root@isvx3 bin]#