cksum cksum cksum!!!

When downloading Oracle binaries across machines using ftp, there are couple of things that you might want to pay attention to get the checksum right.
1. ftp in binary mode.
2. make sure the ulimit is set correctly.

Oracle Database 11g Release 2 (11.2.0.1.0) for AIX (PPC64)
		aix.ppc64_11gR2_database_1of2.zip (1,564,425,851 bytes) (cksum - 1915658395)
		aix.ppc64_11gR2_database_2of2.zip (1,007,010,341 bytes) (cksum - 1152318705)

eg:

isvp18> ftp isvx7.storage.tucson.ibm.com
Connected to isvx7.storage.tucson.ibm.com.
220 (vsFTPd 2.0.5)
Name (isvx7.storage.tucson.ibm.com:root): mayur
331 Please specify the password.
Password:
230 Login successful.
ftp> binary
200 Switching to Binary mode.
ftp> get aix.ppc64_11gR2_database_1of2.zip
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for aix.ppc64_11gR2_database_1of2.zip (1564425851 bytes).
aix.ppc64_11gR2_database_1of2.zip: short write
No space left on device/filesize limit exceeded.
226 File send OK.
500 Unknown command.
1073742400 bytes received in 9.44 seconds (1.111e+05 Kbytes/s)
local: aix.ppc64_11gR2_database_1of2.zip remote: aix.ppc64_11gR2_database_1of2.zip
ftp> get aix.ppc64_11gR2_database_2of2.zip
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for aix.ppc64_11gR2_database_2of2.zip (1007010341 bytes).
226 File send OK.
1007010341 bytes received in 9.018 seconds (1.091e+05 Kbytes/s)
local: aix.ppc64_11gR2_database_2of2.zip remote: aix.ppc64_11gR2_database_2of2.zip
ftp> quit
221 Goodbye. 

isvp18> cksum aix.ppc64_11gR2_database_1of2.zip
416192092 1073741312 aix.ppc64_11gR2_database_1of2.zip
isvp18> ulimit -a
time(seconds)        unlimited
file(blocks)         2097151
data(kbytes)         131072
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     2097151
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user)  unlimited
isvp18> ulimit -c unlimited
isvp18> ulimit -n unlimited
isvp18> ulimit -d unlimited
isvp18> ulimit -f unlimited
isvp18> ulimit -a
time(seconds)        unlimited
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     unlimited
nofiles(descriptors) unlimited
threads(per process) unlimited
processes(per user)  unlimited
isvp18> rm *.zip
rm: Remove aix.ppc64_11gR2_database_1of2.zip? yes
rm: Remove aix.ppc64_11gR2_database_2of2.zip? yes 

isvp18> ftp isvx7.storage.tucson.ibm.com
Connected to isvx7.storage.tucson.ibm.com.
220 (vsFTPd 2.0.5)
Name (isvx7.storage.tucson.ibm.com:root): mayur
331 Please specify the password.
Password:
230 Login successful.
ftp> binary
200 Switching to Binary mode.
ftp> get aix.ppc64_11gR2_database_1of2.zip
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for aix.ppc64_11gR2_database_1of2.zip (1564425851 bytes).
226 File send OK.
1564425851 bytes received in 13.64 seconds (1.12e+05 Kbytes/s)
local: aix.ppc64_11gR2_database_1of2.zip remote: aix.ppc64_11gR2_database_1of2.zip
ftp> get aix.ppc64_11gR2_database_2of2.zip
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for aix.ppc64_11gR2_database_2of2.zip (1007010341 bytes).
226 File send OK.
1007010341 bytes received in 8.968 seconds (1.097e+05 Kbytes/s)
local: aix.ppc64_11gR2_database_2of2.zip remote: aix.ppc64_11gR2_database_2of2.zip
ftp> quit
221 Goodbye.
isvp18> cksum aix.ppc64_11gR2_database_1of2.zip
1915658395 1564425851 aix.ppc64_11gR2_database_1of2.zip
isvp18> cksum aix.ppc64_11gR2_database_2of2.zip
1152318705 1007010341 aix.ppc64_11gR2_database_2of2.zip
isvp18>

Checklist for installing Oracle 11g R2 on AIX 7.1

Checking the Hardware requirements for installing Oracle 11g R2 on AIX 7.1

Memory requirements: Oracle needs at least 4 GB of RAM, and yes we have 24 GB so we are good on that.

isvp17> lsattr -E -l sys0 -a realmem
realmem 25165824 Amount of usable physical memory in Kbytes False

Swap requirements: For 4 GB to 8 GB Memory the swap requirement is 2 times the size of RAM, for 8 GB to 32 GB the requirement is 1.5 times the size of RAM, and if the Memory is more than 32 GB then the swap space required is 32 GB.

isvp17> lsps -a
Page Space      Physical Volume   Volume Group Size %Used Active  Auto  Type Chksum
hd6             hdisk0            rootvg         512MB     3   yes   yes    lv     0

Ok, we not doing good in that front. We'll have to find a way to increase the size of swap from 512MB to
36GB.So let us see how this can be done on AIX 7.1 . There we go, I just added 3

isvp17> lsps -a
Page Space      Physical Volume   Volume Group Size %Used Active  Auto  Type Chksum
paging00        hdisk5            swap         36096MB     1   yes   yes    lv     0
hd6             hdisk0            rootvg         512MB     3   yes   yes    lv     0

Hardware mode: What next, ok we need to know what our hardware is in 32-bit or 64-bit mode.

isvp17> /usr/bin/getconf HARDWARE_BITMODE
64

/tmp size: Now what, yeah what is the disk space requirement. First off /tmp needs to be at least 1 GB. We are ok
with that.

isvp17> df -m /tmp
Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd3        3584.00   3581.39    1%       41     1% /tmp

So what is the space required for the Oracle binaries, and it's datafiles? For Enterprise Edition 7.44 GB (just say 8 GB) is required, and for the data files require 1.55 GB ( you mean 2 GB).

Run Level: Oracle also needs the system run level to be 2. So, how do we find that in AIX

isvp17> who -r
.        run-level 2 Jan 28 11:38       2    0    S

Now that we are done with checking to see if our system meets the hardware requirements for Oracle, let us go ahead and check to see

Service Pack: The OS level needs to be AIX 7.1 with SP 1 or above, with 64-bit kernel. What I currently have is the base AIX 7.1 without the Service Pack, so I downloaded SP2 from http://www-933.ibm.com/support/fixcentral/

isvp17> oslevel -s
7100-00-00-0000

After the installation of the Service Pack(SP2) the oslevel is as follows:

isvp17> oslevel -s
7100-00-02-1041

Additional filesets: Next we have to check if the following filesets are installed Hmmm, it looks like one guy is missing. Can we do without him, I don't know.

isvp17> lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.perfstat \
> bos.perf.libperfstat bos.perf.proctools
Fileset                      Level  State      Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
bos.adt.base               7.1.0.1  COMMITTED  Base Application Development
Toolkit
bos.adt.lib                7.1.0.0  COMMITTED  Base Application Development
Libraries
bos.perf.libperfstat       7.1.0.1  COMMITTED  Performance Statistics Library
Interface
bos.perf.perfstat          7.1.0.0  COMMITTED  Performance Statistics
Interface
bos.perf.proctools         7.1.0.0  COMMITTED  Proc Filesystem Tools

Path: /etc/objrepos
bos.adt.base               7.1.0.0  COMMITTED  Base Application Development
Toolkit
bos.perf.libperfstat       7.1.0.0  COMMITTED  Performance Statistics Library
Interface
bos.perf.perfstat          7.1.0.0  COMMITTED  Performance Statistics
Interface
lslpp: 0504-132  Fileset bos.adt.libm not installed.
isvp17>

Ok, went on to install bos.adt.libm from the original AIX 7.1 base dvd.

isvp17> lslpp -l bos.adt.libm
Fileset                      Level  State      Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
bos.adt.libm               7.1.0.0  COMMITTED  Base Application Development
Math Library

Once the above is done, go and install SP1 or later service pack. In my case I installed SP2 which I had saved under /usr/sys/inst.images

AIX maintainance packages are available from the following web site:
http://www-933.ibm.com/support/fixcentral/

Kernel mode: Next find if the kernel mode is 64

isvp17> getconf KERNEL_BITMODE
64

Compiler: There are some compiler requirements for installing Oracle. I installed the The minimum recommended runtime environment:
IBM XL C/C++ for AIX, V10.1 Runtime Environment
http://www-01.ibm.com/support/docview.wss?rs=2239&uid=swg24019829

UDP and TCP Kernel Parameters:
Use NDD to ensure that the kernel TCP/IP ephemeral port range parameters are set to 9000-65500.

Let me check to see what my current values are:

isvp17> /usr/sbin/no -a | fgrep ephemeral
tcp_ephemeral_high = 65535
tcp_ephemeral_low = 32768
udp_ephemeral_high = 65535
udp_ephemeral_low = 32768

Ok, I need this to bumped.

isvp17> /usr/sbin/no -o tcp_ephemeral_low=9000 -o tcp_ephemeral_high=65500
Setting tcp_ephemeral_low to 9000
Setting tcp_ephemeral_high to 65500
isvp17> /usr/sbin/no -o udp_ephemeral_low=9000 -o udp_ephemeral_high=65500
Setting udp_ephemeral_low to 9000
Setting udp_ephemeral_high to 65500
isvp17>

The options you set with no must be done each time the system starts up. One way to do that is to edit /etc/rc.tcpip and insert your no commands just before the script starts running the server daemons.

Enable core file creation:

isvp17> ulimit -a
time(seconds)        unlimited
file(blocks)         2097151
data(kbytes)         131072
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     2097151
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user)  unlimited
isvp17> ulimit -c unlimited  #turn on corefiles with unlimited size
isvp17> ulimit -n unlimited  #allows an unlimited number of open file descriptors
isvp17> ulimit -d unlimited  #sets the user data limit to unlimited
isvp17> ulimit -f unlimited  #sets the file limit to unlimited
isvp17> ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     unlimited
nofiles(descriptors) unlimited
threads(per process) unlimited
processes(per user)  unlimited

These values are the "soft" limit, and are applied for each user. These values cannot exceed the "hard" limit value. To display and change the hard limits, you can run the ulimit commands using the additional -H command-line option.

Creating users and groups:

isvp17> mkgroup -'A' id='1000' adms='root' oinstall
isvp17> mkgroup -'A' id='2000' adms='root' dba
isvp17> mkgroup -'A' id='4000' adms='root' oper
isvp17> mkuser id='1101' pgrp='oinstall' groups='dba,oper' home='/home/oracle' oracle                        

Create ORACLE_HOME: Create a directory (ORACLE_HOME), for the Oracle Database software.

isvp17> mkdir –p /u01/app/112/dbhome
isvp17> chown –R oracle:oinstall /u01/app/112/dbhome

Run rootpre.sh script: run the rootper.sh script as user root.

 
# cd /home/oracle/database
# ls
doc           response      rootpre.sh    runInstaller  stage
install       rootpre       rpm           sshsetup      welcome.html
# ./rootpre.sh
./rootpre.sh output will be logged in /tmp/rootpre.out_11-02-01.10:21:55
Saving the original files in /etc/ora_save_11-02-01.10:21:55....
Copying new kernel extension to /etc....
Loading the kernel extension from /etc

 Oracle Kernel Extension Loader for AIX
       Copyright (c) 1998,1999 Oracle Corporation


 Successfully loaded /etc/pw-syscall.64bit_kernel with kmid: 0x50a96000
 Successfully configured /etc/pw-syscall.64bit_kernel with kmid: 0x50a96000
The kernel extension was successfuly loaded.

Checking if group services should be configured....
Nothing to configure.
#
#

Mounting a file system on AIX 7.1

I created a 10 GB volume on my Storwize V7000 storage, which I then presented to my AIX 7.1 server.

isvp17> lspv hdisk0          00f65d51a5aa3cf1                    rootvg          active hdisk1          00f65d51bfba4e2e                    test1           active hdisk2          none                                None hdisk3          none                                None hdisk4          none                                None isvp17>

hdisk4 is the 10 GB volume that I had created.Next, I'll create a volume group called "metro".

isvp17> mkvg -y metro hdisk4 0516-1254 mkvg: Changing the PVID in the ODM. metro isvp17> isvp17> lspv hdisk0          00f65d51a5aa3cf1                    rootvg          active hdisk1          00f65d51bfba4e2e                    test1           active hdisk2          none                                None hdisk3          none                                None hdisk4          00f65d51c465f8eb                    metro           active isvp17>

Create a file system using the "smitty crjfs" command. Next select "Add a Standard Journaled File System" from the menu. It will ask you what Volume Group you want the to use, at that point select the volume group "metro" that we had created earlier.

The next screen will ask us a bunch of questions, when when answered correctly created the file system for us. All we have to do after that is mount our newly created file system.

How to increase the file system size on an AIX 7.1 server

I was running out of filesystem space on my AIX 7.1 box

The df output showed that things were pretty tight:

isvp17> df -m
Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4         512.00    330.59   36%     9762    12% /
/dev/hd2       16896.00  14787.64   13%    43253     2% /usr
/dev/hd9var      512.00    232.75   55%     5726    10% /var
/dev/hd3        3584.00   3581.39    1%       40     1% /tmp
/dev/hd1         256.00    120.70   53%      183     1% /home
/dev/hd11admin    256.00    255.62    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt     512.00    336.83   35%     7007     9% /opt
/dev/livedump    256.00    255.64    1%        4     1% /var/adm/ras/livedump
isvp17>

The volume group rootvg in my case had one physical volume(hdisk0). All the physical volumes in
volume group are divided into physical partitions(PPs) of the same size.
To check if there is any free space on the physical volume which has rootvg i did

isvp17> lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            559         454         111..97..22..112..112

454 = number of free physical partitions(PP).

isvp17> chfs -a size=5G /
Filesystem size changed to 10485760
isvp17> df -m
Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4        5120.00   4937.89    4%     9762     1% /
/dev/hd2       16896.00  14787.64   13%    43253     2% /usr
/dev/hd9var      512.00    232.68   55%     5726    10% /var
/dev/hd3        3584.00   3581.39    1%       40     1% /tmp
/dev/hd1         256.00    120.68   53%      183     1% /home
/dev/hd11admin    256.00    255.62    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt     512.00    336.83   35%     7007     9% /opt
/dev/livedump    256.00    255.64    1%        4     1% /var/adm/ras/livedump
isvp17>
isvp17> lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            559         436         111..97..04..112..112
isvp17> 

Next I increased the size of /home filesysem to 15GB

isvp17> chfs -a size=15G /home
Filesystem size changed to 31457280
isvp17> lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            559         377         111..42..00..112..112
isvp17>

It's good to know that I still have 377 physical partitions free in case I need

to increase it further in the future. 

Peek into AIX 7.1

I've been mainly a Linux and Solaris guy for most of my tech career, and now I finally have a chance to tinker
around with AIX, AIX 7.1 to be precise.

AIX Version 7
Copyright IBM Corporation, 1982, 2010.
login: root
root's Password:
*******************************************************************************
*                                                                             *
*                                                                             *
*  Welcome to AIX Version 7.1!                                                *
*                                                                             *
*                                                                             *
*  Please see the README file in /usr/lpp/bos for information pertinent to    *
*  this release of the AIX Operating System.                                  *
*                                                                             *
*                                                                             *
*******************************************************************************
Last login: Fri Jan 21 11:01:47 MST 2011 on /dev/vty0

AIX Level is: 7.1.0.0
isvp17>

Yay!!! that wasn't that hard.


Ok, so lets see what uname gives us here. As you see it tells us the OS name, version, and that we are on
a powerpc system

isvp17> uname -ap
AIX isvp17 1 7 00F65D514C00 powerpc

Next, how about telling us the amount of physical memory on this system. rmss with the -p flag tells us that we have around 24 GB of Memory that has been allocated to this system/LPAR.

isvp17> rmss -p
Simulated memory size is 24576 Mb.

I would also like to know if this a LPAR or if it's a fully allocated system. prtconf will give me a detailed
list of the hardware configuration. Wow!!! this is actually pretty detailed, and handy information to have about
the system that we are working on.

isvp17> prtconf
System Model: IBM,8233-E8B
Machine Serial Number: 065D51P
Processor Type: PowerPC_POWER7
Processor Implementation Mode: POWER 7
Processor Version: PV_7_Compat
Number Of Processors: 6
Processor Clock Speed: 3000 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 1 isvp17
Memory Size: 24576 MB
Good Memory Size: 24576 MB
Platform Firmware level: AL710_099
Firmware Version: IBM,AL710_099
Console Login: enable
Auto Restart: true
Full Core: false

Network Information
        Host Name: isvp17.storage.tucson.ibm.com
        IP Address: x.xx.xx.xx
        Sub Netmask: 255.255.254.0
        Gateway: x.xx.xx.x
        Name Server:
        Domain Name:

Paging Space Information
        Total Paging Space: 512MB
        Percent Used: 3%

Volume Groups Information
==============================================================================
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            559         454         111..97..22..112..112
==============================================================================

INSTALLED RESOURCE LIST

The following resources are installed on the machine.
+/- = Added or deleted from Resource List.
*   = Diagnostic support not available.

  Model Architecture: chrp
  Model Implementation: Multiple Processor, PCI bus

+ sys0                                                           System Object
+ sysplanar0                                                     System Planar
* vio0                                                           Virtual I/O Bus
* ent1             U8233.E8B.065D51P-V1-C3-T1                    Virtual I/O Ethernet Adapter (l-lan)
* vscsi0           U8233.E8B.065D51P-V1-C2-T1                    Virtual SCSI Client Adapter
* hdisk0           U8233.E8B.065D51P-V1-C2-T1-L8100000000000000  Virtual SCSI Disk Drive
* vsa0             U8233.E8B.065D51P-V1-C0                       LPAR Virtual Serial Adapter
* vty0             U8233.E8B.065D51P-V1-C0-L0                    Asynchronous Terminal
+ fcs0             U8233.E8B.065D51P-V1-C36-T1                   Virtual Fibre Channel Client Adapter
+ fscsi0           U8233.E8B.065D51P-V1-C36-T1                   FC SCSI I/O Controller Protocol Device
* lhea0            U78A0.001.DNWK129-P1                          Logical Host Ethernet Adapter (l-hea)
+ ent0             U78A0.001.DNWK129-P1-C6-T1                    Logical Host Ethernet Port (lp-hea)
+ L2cache0                                                       L2 Cache
+ mem0                                                           Memory
+ proc0                                                          Processor
+ proc4                                                          Processor
+ proc8                                                          Processor
+ proc12                                                         Processor
+ proc16                                                         Processor
+ proc20                                                         Processor
isvp17>

How about some information on the disks and the file systems mounted. Let me try out good old df command. Lets see what it brings back.

isvp17> df -m
Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4         512.00    330.74   36%     9749    12% /
/dev/hd2       16896.00  15105.72   11%    43011     2% /usr
/dev/hd9var      512.00    245.90   52%     5684    10% /var
/dev/hd3        3584.00   3581.45    1%       36     1% /tmp
/dev/hd1         256.00    255.54    1%       17     1% /home
/dev/hd11admin    256.00    255.62    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt     512.00    336.88   35%     7007     9% /opt
/dev/livedump    256.00    255.64    1%        4     1% /var/adm/ras/livedump
vanhalen:/vanhalen/tools    512.00    486.43    5%      541     1% /testlab/tools

To mount a filesystem in Solaris you would make an entry into /etc/vfstab , and in RedHat Linux into /etc/fstab.
The non of these file exist in AIX, out here it's dome through the /etc/filesystems file. Lets see what it has.
I won't cut paste the whole thing, but it has someting like this.

isvp17> cat /etc/filesystems
* This version of /etc/filesystems assumes that only the root file system
* is created and ready.  As new file systems are added, change the check,
* mount, free, log, vol and vfs entries for the appropriate stanza.
*

/:
        dev             = /dev/hd4
        vfs             = jfs2
        log             = /dev/hd8
        mount           = automatic
        check           = false
        type            = bootfs
        vol             = root
        free            = true

/home:
        dev             = /dev/hd1
        vfs             = jfs2
        log             = /dev/hd8
        mount           = true
        check           = true
        vol             = /home
        free            = false

/usr:
        dev             = /dev/hd2
        vfs             = jfs2
        log             = /dev/hd8
        mount           = automatic
        check           = false
        type            = bootfs
        vol             = /usr
        free            = false

How about some disk information, something like fdisk -l in Linux, and format in Solaris. Out in AIX land the command is lspv. Ok, so let try it out. Well, I would like some more information that that. Let me look to see if there are any flags that can give me a bit more detailed information. Looks like the -p give us
more info, but it's very different to what I'm used to in Linux and Solaris.

isvp17> lspv
hdisk0          00f65d51a5aa3cf1                    rootvg          active


isvp17> lspv -p hdisk0
hdisk0:
PP RANGE  STATE   REGION        LV NAME             TYPE       MOUNT POINT
  1-1     used    outer edge    hd5                 boot       N/A
  2-112   free    outer edge
113-114   used    outer middle  hd6                 paging     N/A
115-126   used    outer middle  lg_dumplv           sysdump    N/A
127-127   used    outer middle  livedump            jfs2       /var/adm/ras/livedump
128-224   free    outer middle
225-225   used    center        hd8                 jfs2log    N/A
226-226   used    center        hd4                 jfs2       /
227-230   used    center        hd2                 jfs2       /usr
231-231   used    center        hd9var              jfs2       /var
232-232   used    center        hd3                 jfs2       /tmp
233-233   used    center        hd1                 jfs2       /home
234-234   used    center        hd10opt             jfs2       /opt
235-235   used    center        hd11admin           jfs2       /admin
236-236   used    center        hd4                 jfs2       /
237-240   used    center        hd2                 jfs2       /usr
241-241   used    center        hd9var              jfs2       /var
242-242   used    center        hd10opt             jfs2       /opt
243-243   used    center        hd3                 jfs2       /tmp
244-301   used    center        hd2                 jfs2       /usr
302-313   used    center        hd3                 jfs2       /tmp
314-335   free    center
336-447   free    inner middle
448-559   free    inner edge
isvp17>

isvp17> lspv -l hdisk0
hdisk0:
LV NAME               LPs     PPs     DISTRIBUTION          MOUNT POINT
hd3                   14      14      00..00..14..00..00    /tmp
hd9var                2       2       00..00..02..00..00    /var
hd2                   66      66      00..00..66..00..00    /usr
hd4                   2       2       00..00..02..00..00    /
hd10opt               2       2       00..00..02..00..00    /opt
hd1                   1       1       00..00..01..00..00    /home
hd8                   1       1       00..00..01..00..00    N/A
hd6                   2       2       00..02..00..00..00    N/A
hd5                   1       1       01..00..00..00..00    N/A
lg_dumplv             12      12      00..12..00..00..00    N/A
livedump              1       1       00..01..00..00..00    /var/adm/ras/livedump
hd11admin             1       1       00..00..01..00..00    /admin

Backup a MySQL database using IBM Storwize V7000's FlashCopy

I wanted to backup my MySQL databases data dir using the Storwize V7000's FlashCopy functionality.
I first created a small database called address_book, and then added a table called address to it. Next I entered some data into the table address.

[root@isvx3 ~]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.5.8 MySQL Community Server (GPL)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.


mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| test               |
+--------------------+
3 rows in set (0.00 sec)

mysql> create database address_book;
Query OK, 1 row affected (0.00 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| address_book       |
| mysql              |
| test               |
+--------------------+
4 rows in set (0.00 sec)


mysql> use address_book;
Database changed
mysql> create table address (id int not null auto_increment, primary key(id), name varchar(30), address varchar(60));
Query OK, 0 rows affected (0.06 sec)


mysql> insert into address (id, name, address) values (1,"amitabh", "mumbai");
Query OK, 1 row affected (0.00 sec)

mysql> insert into address (id, name, address) values (2,"de niro", "hollywood");
Query OK, 1 row affected (0.00 sec)


mysql> select * from address;
+----+---------+-----------+
| id | name    | address   |
+----+---------+-----------+
|  1 | amitabh | mumbai    |
|  2 | de niro | hollywood |
+----+---------+-----------+
2 rows in set (0.00 sec)

I started the Flashcopy(backup) process on the IBM Storwize V7000. The V7000 created another volume, and started the backup of the volume on which the data dir resided.

/dev/mapper/mpath32p1
9.9G 404M 9.0G 5% /home/mysql

The picture below shows how to backup using the GUI of the Storwize V7000.

As you see below, this has started the FlashCopy backup on the Storwize V7000. The FlashCopy has now created another volume, to which it has started copying our original volume which has the data dir for our MySQL database.

We now map the newly created volume to the MySQL server host as shown below.

While the MySQL data dir is being backed up I went back to my database, and inserted another entry into the mysql table.

mysql> insert into address (id, name, address) values (3,"sean connery", "edinburgh");
Query OK, 1 row affected (0.00 sec)

mysql> select * from address;
+----+--------------+-----------+
| id | name         | address   |
+----+--------------+-----------+
|  1 | amitabh      | mumbai    |
|  2 | de niro      | hollywood |
|  3 | sean connery | edinburgh |
+----+--------------+-----------+
3 rows in set (0.00 sec)

mysql>

As you see above I was able to make another entry into the table, which means that the volume that is being
backuped up the Stowize V7000 is still accessible for use by the MySQL database.

Next I want inject a fault onto my original MySQL data dir file system, and then test my backup volume.
Imagine this to be the fault that causes the volume to be unmounted:

[root@isvx3 ~]# umount /home/mysql
[root@isvx3 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             255G   60G  182G  25% /
/dev/sda1              99M   33M   62M  35% /boot
tmpfs                  32G   16G   17G  49% /dev/shm
[root@isvx3 ~]#

Next, when I try to start mysqld I notice that my database "address_book" is not present.

[root@isvx3 ~]# mysql -u root -p

Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.5.8 MySQL Community Server (GPL)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.



mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| test               |
+--------------------+
3 rows in set (0.03 sec)

mysql>

[root@isvx3 ~]# /etc/init.d/mysql stop
Stopping MySQL:                                            [  OK  ]

Edit the data dir entry in the /etc/my.cnf file to point to the backup 
[root@isvx3 ~]# vi /etc/my.cnf
[root@isvx3 ~]# /etc/init.d/mysql start
Starting MySQL:                                            [  OK  ]
[root@isvx3 ~]# ps -ef | grep mysql
root      9154     1  0 14:48 pts/1    00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/home/mysqlbak/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/lib/mysql/mysqld.log --pid-file=/var/lib/mysql/mysqld/mysqld.pid --user=mysql
mysql     9204  9154  0 14:48 pts/1    00:00:00 /usr/libexec/mysqld --basedir=/usr --datadir=/home/mysqlbak/mysql --user=mysql --pid-file=/var/lib/mysql/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock
root      9222  8065  0 14:49 pts/1    00:00:00 grep mysql
[root@isvx3 ~]#


[root@isvx3 ~]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.5.8 MySQL Community Server (GPL)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.


mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| address_book       |
| mysql              |
| test               |
+--------------------+
4 rows in set (0.00 sec)

mysql> use address_book;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from address;
+----+---------+-----------+
| id | name    | address   |
+----+---------+-----------+
|  1 | amitabh | mumbai    |
|  2 | de niro | hollywood |
+----+---------+-----------+
2 rows in set (0.02 sec)

mysql>

As we see we can see the backup of the first two entries on the new volume. The third entry of the table was entered after the backup was started on Storwize V7000, hence we don't see it.

We can now take another back of original volume, but this time it will be a lot faster then the earlier backup as only the new entries will be copied.

The picture below shows how to start the follow on back

The next time mounted the back up file system, and re-started MySQL I could see the up to date entries in the my MySQL table.

Starting MySQL failed while changing datadir from /var/lib/mysql to /root/metro1/mysql

I created a volume on my IBM Storwize V7000, and presented to my host running Red Hat Enterprise Linux Server release 5.5 (Tikanga)

I then mounted an ext3 file system on a 10 GB partition of the volume.
/dev/mapper/mpath32p              19.9G  171M  9.2G   2% /root/metro1

I had initially installed MySql on the Linux host, and the datadir for my installation was /var/lib/mysql
I now thought that it would be a good idea to move my datadir from /var/lib/mysql to /root/metro1

It seemed a rather straight forward thing to do, so I edited /etc/my.cnf file and changed
[mysqld]
#datadir=/var/lib/mysql
datadir=/root/metro1/mysql

changed the ownership of /root/metro1/mysql to mysql
[root@isvx3 lib]# chown mysql:mysql /root/metro1/mysql
[root@isvx3 lib]# chmod 755 /root/metro1/mysql
When I tried to start mysqd as follows, I got the message that it FAILED
[root@isvx3 lib]# /etc/init.d/mysqld start
Timeout error occurred trying to start MySQL Daemon.
Starting MySQL:                                            [FAILED]


In the log file under /var/lib/mysql/mysqld.log I noticed the following message:
InnoDB: Foreign key constraint system tables created
110114 11:23:25  InnoDB: Started; log sequence number 0 0
110114 11:23:25 [ERROR] bdb:  /root/metro1/mysql: Permission denied
110114 11:23:25 [ERROR] bdb:  /root/metro1/mysql/log.0000000001: Permission denied
110114 11:23:25 [ERROR] bdb:  PANIC: Permission denied
110114 11:23:25 [ERROR] bdb:  PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
110114 11:23:25 [ERROR] bdb:  fatal region error detected; run recovery
110114 11:23:25 [ERROR] bdb:  /root/metro1/mysql: Permission denied
110114 11:23:25 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
110114 11:23:25  mysqld ended

I then changed datadir from /var/lib/mysql to /var/lib/test , and made the ownership and permission changes
to /var/lib/test. I then started the mysqld again, this time mysqld started fine.

Everything seemed the same between the /root/metro1/mysql where mysqld failed, and /var/lib/mysql where mysqld started.

On digging further I noticed the following difference in the permission:
[root@isvx3 lib]# ls -l /
drwxr-xr-x  24 root   root  4096 Sep  1 11:16 var
drwxr-x---  31 root   root  4096 Jan 12 17:37 root

Created a new mount point /home/mysql and mounted the file system on it, intead of /root/metro1
Then I updated the /etc/my.cnf file with the new datadir ie. /home/mysql/mysql

/dev/mapper/mpath32p1          9.9G  171M  9.2G   2% /home/mysql

[root@isvx3 ~]# mkdir -p /home/mysql/mysql
[root@isvx3 ~]# chown mysql:mysql /home/mysql/mysql
[root@isvx3 ~]# chmod 755 /home/mysql/mysql
[root@isvx3 ~]#


After when I started mysqld with the below mentioned command, MySQL started fine.
[root@isvx3 ~]# /etc/init.d/mysqld start
Starting MySQL:                                            [  OK  ]

Also, under /home/mysql/mysql I could see the newly created files

[root@isvx3 mysql]# pwd

/home/mysql/mysql

[root@isvx3 mysql]# ls

ibdata1  ib_logfile0  ib_logfile1  mysql  test

[root@isvx3 mysql]# ls -lh

total 21M

-rw-rw---- 1 mysql mysql  10M Jan 14 14:19 ibdata1

-rw-rw---- 1 mysql mysql 5.0M Jan 14 14:19 ib_logfile0

-rw-rw---- 1 mysql mysql 5.0M Jan 14 14:19 ib_logfile1

drwx------ 2 mysql mysql 4.0K Jan 14 14:19 mysql

drwx------ 2 mysql mysql 4.0K Jan 14 14:19 test

[root@isvx3 mysql]#



Moral of the story is that the error stemmed from the permission of the /root director. I didn't to change the permissions of the /root but instead created the datadir under /home/mysql/mysql to fix the issue.