cksum cksum cksum!!!

When downloading Oracle binaries across machines using ftp, there are couple of things that you might want to pay attention to get the checksum right.
1. ftp in binary mode.
2. make sure the ulimit is set correctly.

Oracle Database 11g Release 2 (11.2.0.1.0) for AIX (PPC64)
		aix.ppc64_11gR2_database_1of2.zip (1,564,425,851 bytes) (cksum - 1915658395)
		aix.ppc64_11gR2_database_2of2.zip (1,007,010,341 bytes) (cksum - 1152318705)

eg:

isvp18> ftp isvx7.storage.tucson.ibm.com
Connected to isvx7.storage.tucson.ibm.com.
220 (vsFTPd 2.0.5)
Name (isvx7.storage.tucson.ibm.com:root): mayur
331 Please specify the password.
Password:
230 Login successful.
ftp> binary
200 Switching to Binary mode.
ftp> get aix.ppc64_11gR2_database_1of2.zip
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for aix.ppc64_11gR2_database_1of2.zip (1564425851 bytes).
aix.ppc64_11gR2_database_1of2.zip: short write
No space left on device/filesize limit exceeded.
226 File send OK.
500 Unknown command.
1073742400 bytes received in 9.44 seconds (1.111e+05 Kbytes/s)
local: aix.ppc64_11gR2_database_1of2.zip remote: aix.ppc64_11gR2_database_1of2.zip
ftp> get aix.ppc64_11gR2_database_2of2.zip
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for aix.ppc64_11gR2_database_2of2.zip (1007010341 bytes).
226 File send OK.
1007010341 bytes received in 9.018 seconds (1.091e+05 Kbytes/s)
local: aix.ppc64_11gR2_database_2of2.zip remote: aix.ppc64_11gR2_database_2of2.zip
ftp> quit
221 Goodbye. 

isvp18> cksum aix.ppc64_11gR2_database_1of2.zip
416192092 1073741312 aix.ppc64_11gR2_database_1of2.zip
isvp18> ulimit -a
time(seconds)        unlimited
file(blocks)         2097151
data(kbytes)         131072
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     2097151
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user)  unlimited
isvp18> ulimit -c unlimited
isvp18> ulimit -n unlimited
isvp18> ulimit -d unlimited
isvp18> ulimit -f unlimited
isvp18> ulimit -a
time(seconds)        unlimited
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     unlimited
nofiles(descriptors) unlimited
threads(per process) unlimited
processes(per user)  unlimited
isvp18> rm *.zip
rm: Remove aix.ppc64_11gR2_database_1of2.zip? yes
rm: Remove aix.ppc64_11gR2_database_2of2.zip? yes 

isvp18> ftp isvx7.storage.tucson.ibm.com
Connected to isvx7.storage.tucson.ibm.com.
220 (vsFTPd 2.0.5)
Name (isvx7.storage.tucson.ibm.com:root): mayur
331 Please specify the password.
Password:
230 Login successful.
ftp> binary
200 Switching to Binary mode.
ftp> get aix.ppc64_11gR2_database_1of2.zip
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for aix.ppc64_11gR2_database_1of2.zip (1564425851 bytes).
226 File send OK.
1564425851 bytes received in 13.64 seconds (1.12e+05 Kbytes/s)
local: aix.ppc64_11gR2_database_1of2.zip remote: aix.ppc64_11gR2_database_1of2.zip
ftp> get aix.ppc64_11gR2_database_2of2.zip
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for aix.ppc64_11gR2_database_2of2.zip (1007010341 bytes).
226 File send OK.
1007010341 bytes received in 8.968 seconds (1.097e+05 Kbytes/s)
local: aix.ppc64_11gR2_database_2of2.zip remote: aix.ppc64_11gR2_database_2of2.zip
ftp> quit
221 Goodbye.
isvp18> cksum aix.ppc64_11gR2_database_1of2.zip
1915658395 1564425851 aix.ppc64_11gR2_database_1of2.zip
isvp18> cksum aix.ppc64_11gR2_database_2of2.zip
1152318705 1007010341 aix.ppc64_11gR2_database_2of2.zip
isvp18>

Checklist for installing Oracle 11g R2 on AIX 7.1

Checking the Hardware requirements for installing Oracle 11g R2 on AIX 7.1

Memory requirements: Oracle needs at least 4 GB of RAM, and yes we have 24 GB so we are good on that.

isvp17> lsattr -E -l sys0 -a realmem
realmem 25165824 Amount of usable physical memory in Kbytes False

Swap requirements: For 4 GB to 8 GB Memory the swap requirement is 2 times the size of RAM, for 8 GB to 32 GB the requirement is 1.5 times the size of RAM, and if the Memory is more than 32 GB then the swap space required is 32 GB.

isvp17> lsps -a
Page Space      Physical Volume   Volume Group Size %Used Active  Auto  Type Chksum
hd6             hdisk0            rootvg         512MB     3   yes   yes    lv     0

Ok, we not doing good in that front. We'll have to find a way to increase the size of swap from 512MB to
36GB.So let us see how this can be done on AIX 7.1 . There we go, I just added 3

isvp17> lsps -a
Page Space      Physical Volume   Volume Group Size %Used Active  Auto  Type Chksum
paging00        hdisk5            swap         36096MB     1   yes   yes    lv     0
hd6             hdisk0            rootvg         512MB     3   yes   yes    lv     0

Hardware mode: What next, ok we need to know what our hardware is in 32-bit or 64-bit mode.

isvp17> /usr/bin/getconf HARDWARE_BITMODE
64

/tmp size: Now what, yeah what is the disk space requirement. First off /tmp needs to be at least 1 GB. We are ok
with that.

isvp17> df -m /tmp
Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd3        3584.00   3581.39    1%       41     1% /tmp

So what is the space required for the Oracle binaries, and it's datafiles? For Enterprise Edition 7.44 GB (just say 8 GB) is required, and for the data files require 1.55 GB ( you mean 2 GB).

Run Level: Oracle also needs the system run level to be 2. So, how do we find that in AIX

isvp17> who -r
.        run-level 2 Jan 28 11:38       2    0    S

Now that we are done with checking to see if our system meets the hardware requirements for Oracle, let us go ahead and check to see

Service Pack: The OS level needs to be AIX 7.1 with SP 1 or above, with 64-bit kernel. What I currently have is the base AIX 7.1 without the Service Pack, so I downloaded SP2 from http://www-933.ibm.com/support/fixcentral/

isvp17> oslevel -s
7100-00-00-0000

After the installation of the Service Pack(SP2) the oslevel is as follows:

isvp17> oslevel -s
7100-00-02-1041

Additional filesets: Next we have to check if the following filesets are installed Hmmm, it looks like one guy is missing. Can we do without him, I don't know.

isvp17> lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.perfstat \
> bos.perf.libperfstat bos.perf.proctools
Fileset                      Level  State      Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
bos.adt.base               7.1.0.1  COMMITTED  Base Application Development
Toolkit
bos.adt.lib                7.1.0.0  COMMITTED  Base Application Development
Libraries
bos.perf.libperfstat       7.1.0.1  COMMITTED  Performance Statistics Library
Interface
bos.perf.perfstat          7.1.0.0  COMMITTED  Performance Statistics
Interface
bos.perf.proctools         7.1.0.0  COMMITTED  Proc Filesystem Tools

Path: /etc/objrepos
bos.adt.base               7.1.0.0  COMMITTED  Base Application Development
Toolkit
bos.perf.libperfstat       7.1.0.0  COMMITTED  Performance Statistics Library
Interface
bos.perf.perfstat          7.1.0.0  COMMITTED  Performance Statistics
Interface
lslpp: 0504-132  Fileset bos.adt.libm not installed.
isvp17>

Ok, went on to install bos.adt.libm from the original AIX 7.1 base dvd.

isvp17> lslpp -l bos.adt.libm
Fileset                      Level  State      Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
bos.adt.libm               7.1.0.0  COMMITTED  Base Application Development
Math Library

Once the above is done, go and install SP1 or later service pack. In my case I installed SP2 which I had saved under /usr/sys/inst.images

AIX maintainance packages are available from the following web site:
http://www-933.ibm.com/support/fixcentral/

Kernel mode: Next find if the kernel mode is 64

isvp17> getconf KERNEL_BITMODE
64

Compiler: There are some compiler requirements for installing Oracle. I installed the The minimum recommended runtime environment:
IBM XL C/C++ for AIX, V10.1 Runtime Environment
http://www-01.ibm.com/support/docview.wss?rs=2239&uid=swg24019829

UDP and TCP Kernel Parameters:
Use NDD to ensure that the kernel TCP/IP ephemeral port range parameters are set to 9000-65500.

Let me check to see what my current values are:

isvp17> /usr/sbin/no -a | fgrep ephemeral
tcp_ephemeral_high = 65535
tcp_ephemeral_low = 32768
udp_ephemeral_high = 65535
udp_ephemeral_low = 32768

Ok, I need this to bumped.

isvp17> /usr/sbin/no -o tcp_ephemeral_low=9000 -o tcp_ephemeral_high=65500
Setting tcp_ephemeral_low to 9000
Setting tcp_ephemeral_high to 65500
isvp17> /usr/sbin/no -o udp_ephemeral_low=9000 -o udp_ephemeral_high=65500
Setting udp_ephemeral_low to 9000
Setting udp_ephemeral_high to 65500
isvp17>

The options you set with no must be done each time the system starts up. One way to do that is to edit /etc/rc.tcpip and insert your no commands just before the script starts running the server daemons.

Enable core file creation:

isvp17> ulimit -a
time(seconds)        unlimited
file(blocks)         2097151
data(kbytes)         131072
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     2097151
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user)  unlimited
isvp17> ulimit -c unlimited  #turn on corefiles with unlimited size
isvp17> ulimit -n unlimited  #allows an unlimited number of open file descriptors
isvp17> ulimit -d unlimited  #sets the user data limit to unlimited
isvp17> ulimit -f unlimited  #sets the file limit to unlimited
isvp17> ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     unlimited
nofiles(descriptors) unlimited
threads(per process) unlimited
processes(per user)  unlimited

These values are the "soft" limit, and are applied for each user. These values cannot exceed the "hard" limit value. To display and change the hard limits, you can run the ulimit commands using the additional -H command-line option.

Creating users and groups:

isvp17> mkgroup -'A' id='1000' adms='root' oinstall
isvp17> mkgroup -'A' id='2000' adms='root' dba
isvp17> mkgroup -'A' id='4000' adms='root' oper
isvp17> mkuser id='1101' pgrp='oinstall' groups='dba,oper' home='/home/oracle' oracle                        

Create ORACLE_HOME: Create a directory (ORACLE_HOME), for the Oracle Database software.

isvp17> mkdir –p /u01/app/112/dbhome
isvp17> chown –R oracle:oinstall /u01/app/112/dbhome

Run rootpre.sh script: run the rootper.sh script as user root.

 
# cd /home/oracle/database
# ls
doc           response      rootpre.sh    runInstaller  stage
install       rootpre       rpm           sshsetup      welcome.html
# ./rootpre.sh
./rootpre.sh output will be logged in /tmp/rootpre.out_11-02-01.10:21:55
Saving the original files in /etc/ora_save_11-02-01.10:21:55....
Copying new kernel extension to /etc....
Loading the kernel extension from /etc

 Oracle Kernel Extension Loader for AIX
       Copyright (c) 1998,1999 Oracle Corporation


 Successfully loaded /etc/pw-syscall.64bit_kernel with kmid: 0x50a96000
 Successfully configured /etc/pw-syscall.64bit_kernel with kmid: 0x50a96000
The kernel extension was successfuly loaded.

Checking if group services should be configured....
Nothing to configure.
#
#

Mounting a file system on AIX 7.1

I created a 10 GB volume on my Storwize V7000 storage, which I then presented to my AIX 7.1 server.

isvp17> lspv hdisk0          00f65d51a5aa3cf1                    rootvg          active hdisk1          00f65d51bfba4e2e                    test1           active hdisk2          none                                None hdisk3          none                                None hdisk4          none                                None isvp17>

hdisk4 is the 10 GB volume that I had created.Next, I'll create a volume group called "metro".

isvp17> mkvg -y metro hdisk4 0516-1254 mkvg: Changing the PVID in the ODM. metro isvp17> isvp17> lspv hdisk0          00f65d51a5aa3cf1                    rootvg          active hdisk1          00f65d51bfba4e2e                    test1           active hdisk2          none                                None hdisk3          none                                None hdisk4          00f65d51c465f8eb                    metro           active isvp17>

Create a file system using the "smitty crjfs" command. Next select "Add a Standard Journaled File System" from the menu. It will ask you what Volume Group you want the to use, at that point select the volume group "metro" that we had created earlier.

The next screen will ask us a bunch of questions, when when answered correctly created the file system for us. All we have to do after that is mount our newly created file system.

How to increase the file system size on an AIX 7.1 server

I was running out of filesystem space on my AIX 7.1 box

The df output showed that things were pretty tight:

isvp17> df -m
Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4         512.00    330.59   36%     9762    12% /
/dev/hd2       16896.00  14787.64   13%    43253     2% /usr
/dev/hd9var      512.00    232.75   55%     5726    10% /var
/dev/hd3        3584.00   3581.39    1%       40     1% /tmp
/dev/hd1         256.00    120.70   53%      183     1% /home
/dev/hd11admin    256.00    255.62    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt     512.00    336.83   35%     7007     9% /opt
/dev/livedump    256.00    255.64    1%        4     1% /var/adm/ras/livedump
isvp17>

The volume group rootvg in my case had one physical volume(hdisk0). All the physical volumes in
volume group are divided into physical partitions(PPs) of the same size.
To check if there is any free space on the physical volume which has rootvg i did

isvp17> lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            559         454         111..97..22..112..112

454 = number of free physical partitions(PP).

isvp17> chfs -a size=5G /
Filesystem size changed to 10485760
isvp17> df -m
Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4        5120.00   4937.89    4%     9762     1% /
/dev/hd2       16896.00  14787.64   13%    43253     2% /usr
/dev/hd9var      512.00    232.68   55%     5726    10% /var
/dev/hd3        3584.00   3581.39    1%       40     1% /tmp
/dev/hd1         256.00    120.68   53%      183     1% /home
/dev/hd11admin    256.00    255.62    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt     512.00    336.83   35%     7007     9% /opt
/dev/livedump    256.00    255.64    1%        4     1% /var/adm/ras/livedump
isvp17>
isvp17> lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            559         436         111..97..04..112..112
isvp17> 

Next I increased the size of /home filesysem to 15GB

isvp17> chfs -a size=15G /home
Filesystem size changed to 31457280
isvp17> lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            559         377         111..42..00..112..112
isvp17>

It's good to know that I still have 377 physical partitions free in case I need

to increase it further in the future. 

Peek into AIX 7.1

I've been mainly a Linux and Solaris guy for most of my tech career, and now I finally have a chance to tinker
around with AIX, AIX 7.1 to be precise.

AIX Version 7
Copyright IBM Corporation, 1982, 2010.
login: root
root's Password:
*******************************************************************************
*                                                                             *
*                                                                             *
*  Welcome to AIX Version 7.1!                                                *
*                                                                             *
*                                                                             *
*  Please see the README file in /usr/lpp/bos for information pertinent to    *
*  this release of the AIX Operating System.                                  *
*                                                                             *
*                                                                             *
*******************************************************************************
Last login: Fri Jan 21 11:01:47 MST 2011 on /dev/vty0

AIX Level is: 7.1.0.0
isvp17>

Yay!!! that wasn't that hard.


Ok, so lets see what uname gives us here. As you see it tells us the OS name, version, and that we are on
a powerpc system

isvp17> uname -ap
AIX isvp17 1 7 00F65D514C00 powerpc

Next, how about telling us the amount of physical memory on this system. rmss with the -p flag tells us that we have around 24 GB of Memory that has been allocated to this system/LPAR.

isvp17> rmss -p
Simulated memory size is 24576 Mb.

I would also like to know if this a LPAR or if it's a fully allocated system. prtconf will give me a detailed
list of the hardware configuration. Wow!!! this is actually pretty detailed, and handy information to have about
the system that we are working on.

isvp17> prtconf
System Model: IBM,8233-E8B
Machine Serial Number: 065D51P
Processor Type: PowerPC_POWER7
Processor Implementation Mode: POWER 7
Processor Version: PV_7_Compat
Number Of Processors: 6
Processor Clock Speed: 3000 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 1 isvp17
Memory Size: 24576 MB
Good Memory Size: 24576 MB
Platform Firmware level: AL710_099
Firmware Version: IBM,AL710_099
Console Login: enable
Auto Restart: true
Full Core: false

Network Information
        Host Name: isvp17.storage.tucson.ibm.com
        IP Address: x.xx.xx.xx
        Sub Netmask: 255.255.254.0
        Gateway: x.xx.xx.x
        Name Server:
        Domain Name:

Paging Space Information
        Total Paging Space: 512MB
        Percent Used: 3%

Volume Groups Information
==============================================================================
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            559         454         111..97..22..112..112
==============================================================================

INSTALLED RESOURCE LIST

The following resources are installed on the machine.
+/- = Added or deleted from Resource List.
*   = Diagnostic support not available.

  Model Architecture: chrp
  Model Implementation: Multiple Processor, PCI bus

+ sys0                                                           System Object
+ sysplanar0                                                     System Planar
* vio0                                                           Virtual I/O Bus
* ent1             U8233.E8B.065D51P-V1-C3-T1                    Virtual I/O Ethernet Adapter (l-lan)
* vscsi0           U8233.E8B.065D51P-V1-C2-T1                    Virtual SCSI Client Adapter
* hdisk0           U8233.E8B.065D51P-V1-C2-T1-L8100000000000000  Virtual SCSI Disk Drive
* vsa0             U8233.E8B.065D51P-V1-C0                       LPAR Virtual Serial Adapter
* vty0             U8233.E8B.065D51P-V1-C0-L0                    Asynchronous Terminal
+ fcs0             U8233.E8B.065D51P-V1-C36-T1                   Virtual Fibre Channel Client Adapter
+ fscsi0           U8233.E8B.065D51P-V1-C36-T1                   FC SCSI I/O Controller Protocol Device
* lhea0            U78A0.001.DNWK129-P1                          Logical Host Ethernet Adapter (l-hea)
+ ent0             U78A0.001.DNWK129-P1-C6-T1                    Logical Host Ethernet Port (lp-hea)
+ L2cache0                                                       L2 Cache
+ mem0                                                           Memory
+ proc0                                                          Processor
+ proc4                                                          Processor
+ proc8                                                          Processor
+ proc12                                                         Processor
+ proc16                                                         Processor
+ proc20                                                         Processor
isvp17>

How about some information on the disks and the file systems mounted. Let me try out good old df command. Lets see what it brings back.

isvp17> df -m
Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4         512.00    330.74   36%     9749    12% /
/dev/hd2       16896.00  15105.72   11%    43011     2% /usr
/dev/hd9var      512.00    245.90   52%     5684    10% /var
/dev/hd3        3584.00   3581.45    1%       36     1% /tmp
/dev/hd1         256.00    255.54    1%       17     1% /home
/dev/hd11admin    256.00    255.62    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt     512.00    336.88   35%     7007     9% /opt
/dev/livedump    256.00    255.64    1%        4     1% /var/adm/ras/livedump
vanhalen:/vanhalen/tools    512.00    486.43    5%      541     1% /testlab/tools

To mount a filesystem in Solaris you would make an entry into /etc/vfstab , and in RedHat Linux into /etc/fstab.
The non of these file exist in AIX, out here it's dome through the /etc/filesystems file. Lets see what it has.
I won't cut paste the whole thing, but it has someting like this.

isvp17> cat /etc/filesystems
* This version of /etc/filesystems assumes that only the root file system
* is created and ready.  As new file systems are added, change the check,
* mount, free, log, vol and vfs entries for the appropriate stanza.
*

/:
        dev             = /dev/hd4
        vfs             = jfs2
        log             = /dev/hd8
        mount           = automatic
        check           = false
        type            = bootfs
        vol             = root
        free            = true

/home:
        dev             = /dev/hd1
        vfs             = jfs2
        log             = /dev/hd8
        mount           = true
        check           = true
        vol             = /home
        free            = false

/usr:
        dev             = /dev/hd2
        vfs             = jfs2
        log             = /dev/hd8
        mount           = automatic
        check           = false
        type            = bootfs
        vol             = /usr
        free            = false

How about some disk information, something like fdisk -l in Linux, and format in Solaris. Out in AIX land the command is lspv. Ok, so let try it out. Well, I would like some more information that that. Let me look to see if there are any flags that can give me a bit more detailed information. Looks like the -p give us
more info, but it's very different to what I'm used to in Linux and Solaris.

isvp17> lspv
hdisk0          00f65d51a5aa3cf1                    rootvg          active


isvp17> lspv -p hdisk0
hdisk0:
PP RANGE  STATE   REGION        LV NAME             TYPE       MOUNT POINT
  1-1     used    outer edge    hd5                 boot       N/A
  2-112   free    outer edge
113-114   used    outer middle  hd6                 paging     N/A
115-126   used    outer middle  lg_dumplv           sysdump    N/A
127-127   used    outer middle  livedump            jfs2       /var/adm/ras/livedump
128-224   free    outer middle
225-225   used    center        hd8                 jfs2log    N/A
226-226   used    center        hd4                 jfs2       /
227-230   used    center        hd2                 jfs2       /usr
231-231   used    center        hd9var              jfs2       /var
232-232   used    center        hd3                 jfs2       /tmp
233-233   used    center        hd1                 jfs2       /home
234-234   used    center        hd10opt             jfs2       /opt
235-235   used    center        hd11admin           jfs2       /admin
236-236   used    center        hd4                 jfs2       /
237-240   used    center        hd2                 jfs2       /usr
241-241   used    center        hd9var              jfs2       /var
242-242   used    center        hd10opt             jfs2       /opt
243-243   used    center        hd3                 jfs2       /tmp
244-301   used    center        hd2                 jfs2       /usr
302-313   used    center        hd3                 jfs2       /tmp
314-335   free    center
336-447   free    inner middle
448-559   free    inner edge
isvp17>

isvp17> lspv -l hdisk0
hdisk0:
LV NAME               LPs     PPs     DISTRIBUTION          MOUNT POINT
hd3                   14      14      00..00..14..00..00    /tmp
hd9var                2       2       00..00..02..00..00    /var
hd2                   66      66      00..00..66..00..00    /usr
hd4                   2       2       00..00..02..00..00    /
hd10opt               2       2       00..00..02..00..00    /opt
hd1                   1       1       00..00..01..00..00    /home
hd8                   1       1       00..00..01..00..00    N/A
hd6                   2       2       00..02..00..00..00    N/A
hd5                   1       1       01..00..00..00..00    N/A
lg_dumplv             12      12      00..12..00..00..00    N/A
livedump              1       1       00..01..00..00..00    /var/adm/ras/livedump
hd11admin             1       1       00..00..01..00..00    /admin

Backup a MySQL database using IBM Storwize V7000's FlashCopy

I wanted to backup my MySQL databases data dir using the Storwize V7000's FlashCopy functionality.
I first created a small database called address_book, and then added a table called address to it. Next I entered some data into the table address.

[root@isvx3 ~]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.5.8 MySQL Community Server (GPL)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.


mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| test               |
+--------------------+
3 rows in set (0.00 sec)

mysql> create database address_book;
Query OK, 1 row affected (0.00 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| address_book       |
| mysql              |
| test               |
+--------------------+
4 rows in set (0.00 sec)


mysql> use address_book;
Database changed
mysql> create table address (id int not null auto_increment, primary key(id), name varchar(30), address varchar(60));
Query OK, 0 rows affected (0.06 sec)


mysql> insert into address (id, name, address) values (1,"amitabh", "mumbai");
Query OK, 1 row affected (0.00 sec)

mysql> insert into address (id, name, address) values (2,"de niro", "hollywood");
Query OK, 1 row affected (0.00 sec)


mysql> select * from address;
+----+---------+-----------+
| id | name    | address   |
+----+---------+-----------+
|  1 | amitabh | mumbai    |
|  2 | de niro | hollywood |
+----+---------+-----------+
2 rows in set (0.00 sec)

I started the Flashcopy(backup) process on the IBM Storwize V7000. The V7000 created another volume, and started the backup of the volume on which the data dir resided.

/dev/mapper/mpath32p1
9.9G 404M 9.0G 5% /home/mysql

The picture below shows how to backup using the GUI of the Storwize V7000.

As you see below, this has started the FlashCopy backup on the Storwize V7000. The FlashCopy has now created another volume, to which it has started copying our original volume which has the data dir for our MySQL database.

We now map the newly created volume to the MySQL server host as shown below.

While the MySQL data dir is being backed up I went back to my database, and inserted another entry into the mysql table.

mysql> insert into address (id, name, address) values (3,"sean connery", "edinburgh");
Query OK, 1 row affected (0.00 sec)

mysql> select * from address;
+----+--------------+-----------+
| id | name         | address   |
+----+--------------+-----------+
|  1 | amitabh      | mumbai    |
|  2 | de niro      | hollywood |
|  3 | sean connery | edinburgh |
+----+--------------+-----------+
3 rows in set (0.00 sec)

mysql>

As you see above I was able to make another entry into the table, which means that the volume that is being
backuped up the Stowize V7000 is still accessible for use by the MySQL database.

Next I want inject a fault onto my original MySQL data dir file system, and then test my backup volume.
Imagine this to be the fault that causes the volume to be unmounted:

[root@isvx3 ~]# umount /home/mysql
[root@isvx3 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             255G   60G  182G  25% /
/dev/sda1              99M   33M   62M  35% /boot
tmpfs                  32G   16G   17G  49% /dev/shm
[root@isvx3 ~]#

Next, when I try to start mysqld I notice that my database "address_book" is not present.

[root@isvx3 ~]# mysql -u root -p

Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.5.8 MySQL Community Server (GPL)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.



mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| test               |
+--------------------+
3 rows in set (0.03 sec)

mysql>

[root@isvx3 ~]# /etc/init.d/mysql stop
Stopping MySQL:                                            [  OK  ]

Edit the data dir entry in the /etc/my.cnf file to point to the backup 
[root@isvx3 ~]# vi /etc/my.cnf
[root@isvx3 ~]# /etc/init.d/mysql start
Starting MySQL:                                            [  OK  ]
[root@isvx3 ~]# ps -ef | grep mysql
root      9154     1  0 14:48 pts/1    00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/home/mysqlbak/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/lib/mysql/mysqld.log --pid-file=/var/lib/mysql/mysqld/mysqld.pid --user=mysql
mysql     9204  9154  0 14:48 pts/1    00:00:00 /usr/libexec/mysqld --basedir=/usr --datadir=/home/mysqlbak/mysql --user=mysql --pid-file=/var/lib/mysql/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock
root      9222  8065  0 14:49 pts/1    00:00:00 grep mysql
[root@isvx3 ~]#


[root@isvx3 ~]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.5.8 MySQL Community Server (GPL)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.


mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| address_book       |
| mysql              |
| test               |
+--------------------+
4 rows in set (0.00 sec)

mysql> use address_book;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from address;
+----+---------+-----------+
| id | name    | address   |
+----+---------+-----------+
|  1 | amitabh | mumbai    |
|  2 | de niro | hollywood |
+----+---------+-----------+
2 rows in set (0.02 sec)

mysql>

As we see we can see the backup of the first two entries on the new volume. The third entry of the table was entered after the backup was started on Storwize V7000, hence we don't see it.

We can now take another back of original volume, but this time it will be a lot faster then the earlier backup as only the new entries will be copied.

The picture below shows how to start the follow on back

The next time mounted the back up file system, and re-started MySQL I could see the up to date entries in the my MySQL table.

Starting MySQL failed while changing datadir from /var/lib/mysql to /root/metro1/mysql

I created a volume on my IBM Storwize V7000, and presented to my host running Red Hat Enterprise Linux Server release 5.5 (Tikanga)

I then mounted an ext3 file system on a 10 GB partition of the volume.
/dev/mapper/mpath32p              19.9G  171M  9.2G   2% /root/metro1

I had initially installed MySql on the Linux host, and the datadir for my installation was /var/lib/mysql
I now thought that it would be a good idea to move my datadir from /var/lib/mysql to /root/metro1

It seemed a rather straight forward thing to do, so I edited /etc/my.cnf file and changed
[mysqld]
#datadir=/var/lib/mysql
datadir=/root/metro1/mysql

changed the ownership of /root/metro1/mysql to mysql
[root@isvx3 lib]# chown mysql:mysql /root/metro1/mysql
[root@isvx3 lib]# chmod 755 /root/metro1/mysql
When I tried to start mysqd as follows, I got the message that it FAILED
[root@isvx3 lib]# /etc/init.d/mysqld start
Timeout error occurred trying to start MySQL Daemon.
Starting MySQL:                                            [FAILED]


In the log file under /var/lib/mysql/mysqld.log I noticed the following message:
InnoDB: Foreign key constraint system tables created
110114 11:23:25  InnoDB: Started; log sequence number 0 0
110114 11:23:25 [ERROR] bdb:  /root/metro1/mysql: Permission denied
110114 11:23:25 [ERROR] bdb:  /root/metro1/mysql/log.0000000001: Permission denied
110114 11:23:25 [ERROR] bdb:  PANIC: Permission denied
110114 11:23:25 [ERROR] bdb:  PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
110114 11:23:25 [ERROR] bdb:  fatal region error detected; run recovery
110114 11:23:25 [ERROR] bdb:  /root/metro1/mysql: Permission denied
110114 11:23:25 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
110114 11:23:25  mysqld ended

I then changed datadir from /var/lib/mysql to /var/lib/test , and made the ownership and permission changes
to /var/lib/test. I then started the mysqld again, this time mysqld started fine.

Everything seemed the same between the /root/metro1/mysql where mysqld failed, and /var/lib/mysql where mysqld started.

On digging further I noticed the following difference in the permission:
[root@isvx3 lib]# ls -l /
drwxr-xr-x  24 root   root  4096 Sep  1 11:16 var
drwxr-x---  31 root   root  4096 Jan 12 17:37 root

Created a new mount point /home/mysql and mounted the file system on it, intead of /root/metro1
Then I updated the /etc/my.cnf file with the new datadir ie. /home/mysql/mysql

/dev/mapper/mpath32p1          9.9G  171M  9.2G   2% /home/mysql

[root@isvx3 ~]# mkdir -p /home/mysql/mysql
[root@isvx3 ~]# chown mysql:mysql /home/mysql/mysql
[root@isvx3 ~]# chmod 755 /home/mysql/mysql
[root@isvx3 ~]#


After when I started mysqld with the below mentioned command, MySQL started fine.
[root@isvx3 ~]# /etc/init.d/mysqld start
Starting MySQL:                                            [  OK  ]

Also, under /home/mysql/mysql I could see the newly created files

[root@isvx3 mysql]# pwd

/home/mysql/mysql

[root@isvx3 mysql]# ls

ibdata1  ib_logfile0  ib_logfile1  mysql  test

[root@isvx3 mysql]# ls -lh

total 21M

-rw-rw---- 1 mysql mysql  10M Jan 14 14:19 ibdata1

-rw-rw---- 1 mysql mysql 5.0M Jan 14 14:19 ib_logfile0

-rw-rw---- 1 mysql mysql 5.0M Jan 14 14:19 ib_logfile1

drwx------ 2 mysql mysql 4.0K Jan 14 14:19 mysql

drwx------ 2 mysql mysql 4.0K Jan 14 14:19 test

[root@isvx3 mysql]#



Moral of the story is that the error stemmed from the permission of the /root director. I didn't to change the permissions of the /root but instead created the datadir under /home/mysql/mysql to fix the issue.

What does /etc/init.d/mysqld start really do?

Well I started mysql using the following
# /etc/init.d/mysqld start

First let us see what the script /etc/init.d/mysqld does when "start" is passed to it.

1. It creates the error log file /var/log/mysqld.log

2. Changes the permission of the owner and group to "mysql"

3. If /var/lib/mysql/mysql is not present, then initialize the database by running the script /usr/bin/mysql_install_db

4. Change the user and group of /var/lib/mysql to "mysql", and the permission of it to 0755

5. Next call /usr/bin/mysqld_safe and pass it the data directory, socket file name, error log file, the pid file name, and the user name.

6. Finally check if the server has come up by checking the response /usr/bin/mysqladmin

MySQL installation on Red Hat Enterprise Linux is yummy!!!

Login as root onto the Red Hat Enterprise Linux server

[root@isvx3 ~]# yum install mysql-server mysql
Loaded plugins: rhnplugin, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package mysql.i386 0:5.0.77-4.el5_5.4 set to be updated
--> Processing Dependency: perl(DBI) for package: mysql
---> Package mysql.x86_64 0:5.0.77-4.el5_5.4 set to be updated
---> Package mysql-server.x86_64 0:5.0.77-4.el5_5.4 set to be updated
--> Processing Dependency: perl-DBD-MySQL for package: mysql-server
--> Running transaction check
---> Package perl-DBD-MySQL.x86_64 0:3.0007-2.el5 set to be updated
---> Package perl-DBI.x86_64 0:1.52-2.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package           Arch      Version              Repository               Size
================================================================================
Installing:
 mysql             i386      5.0.77-4.el5_5.4     rhel-x86_64-server-5    4.8 M
 mysql             x86_64    5.0.77-4.el5_5.4     rhel-x86_64-server-5    4.8 M
 mysql-server      x86_64    5.0.77-4.el5_5.4     rhel-x86_64-server-5    9.8 M
Installing for dependencies:
 perl-DBD-MySQL    x86_64    3.0007-2.el5         rhel-x86_64-server-5    147 k
 perl-DBI          x86_64    1.52-2.el5           rhel-x86_64-server-5    605 k

Transaction Summary
================================================================================
Install       5 Package(s)
Upgrade       0 Package(s)

Total download size: 20 M
Is this ok [y/N]: y
Downloading Packages:
(1/5): perl-DBD-MySQL-3.0007-2.el5.x86_64.rpm            | 147 kB     00:00
(2/5): perl-DBI-1.52-2.el5.x86_64.rpm                    | 605 kB     00:00
(3/5): mysql-5.0.77-4.el5_5.4.i386.rpm                   | 4.8 MB     00:02
(4/5): mysql-5.0.77-4.el5_5.4.x86_64.rpm                                                                                                             | 4.8 MB     00:15
(5/5): mysql-server-5.0.77-4.el5_5.4.x86_64.rpm                                                                                                      | 9.8 MB     00:10
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                       657 kB/s |  20 MB     00:31
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : perl-DBI                                                                                                                                             1/5
  Installing     : mysql                                                                                                                                                2/5
  Installing     : perl-DBD-MySQL                                                                                                                                       3/5
  Installing     : mysql                                                                                                                                                4/5
  Installing     : mysql-server                                                                                                                                         5/5

Installed:
  mysql.i386 0:5.0.77-4.el5_5.4                        mysql.x86_64 0:5.0.77-4.el5_5.4                        mysql-server.x86_64 0:5.0.77-4.el5_5.4

Dependency Installed:
  perl-DBD-MySQL.x86_64 0:3.0007-2.el5                                                     perl-DBI.x86_64 0:1.52-2.el5

Complete!
[root@isvx3 ~]#

The installation created user "mysql" in the /etc/passwd file

-bash-3.2$ cat /etc/passwd | grep mysql
mysql:x:27:27:MySQL Server:/var/lib/mysql:/bin/bash

[root@isvx3 ~]# su - mysql
-bash-3.2$ pwd
/var/lib/mysql
 -bash-3.2$ id
uid=27(mysql) gid=27(mysql) groups=27(mysql)

Now that we have completed the installation bits, let us try and start it.

[root@isvx3 etc]# ps -ef | grep mysql
root      2560 21023  0 17:13 pts/1    00:00:00 grep mysql
[root@isvx3 etc]# chkconfig mysqld on
[root@isvx3 etc]# /etc/init.d/mysqld start
Initializing MySQL database:  Installing MySQL system tables...
OK
Filling help tables...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:
/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h isvx3.storage.tucson.ibm.com password 'new-password'

Alternatively you can run:
/usr/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd mysql-test ; perl mysql-test-run.pl

Please report any problems with the /usr/bin/mysqlbug script!

The latest information about MySQL is available on the web at
http://www.mysql.com
Support MySQL by buying support/licenses at http://shop.mysql.com
                                                           [  OK  ]
Starting MySQL:                                            [  OK  ]

Now let us check the MySQL processes on the system

[root@isvx3 etc]# ps -ef | grep mysql
root      2929     1  0 17:14 pts/1    00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --user=mysql
mysql     2979  2929  0 17:14 pts/1    00:00:00 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock
root      3188 21023  0 17:14 pts/1    00:00:00 grep sql
[root@isvx3 etc]#

Everything you wanted to know about your HBA but were afraid to ask!!!

Well, as usual I was provided with a remote access to our lab machines at the beginning of the project, but needed to collect a whole lot of information about the server during the course of the project.

One such piece of information was about the HBA on the RedHat Linux box that I was working on.

You can find all the information you need about the HBA under /sys/class/fc_host

[root@isvx7 host5]# pwd
/sys/class/fc_host

[root@isvx7 fc_host]# ls
host5  host6

[root@isvx7 host5]# ls
device       port_id     speed              supported_speeds  uevent
fabric_name  port_name   statistics         symbolic_name
issue_lip    port_state  subsystem          system_hostname
node_name    port_type   supported_classes  tgtid_bind_type
[root@isvx7 host5]# cat speed
8 Gbit

[root@isvx7 host5]# cat supported_speeds
1 Gbit, 2 Gbit, 4 Gbit, 8 Gbit

[root@isvx7 host5]# lspci |grep -i -e fib -e hba
15:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
15:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
[root@isvx7 host5]#


[root@isvx7 proc]# systool -c fc_host -v
Class = "fc_host"

Class Device = "host5"
Class Device path = "/sys/class/fc_host/host5"
fabric_name         = "0x100000051ef510a4"
issue_lip           = 
node_name           = "0x20000024ff21d2ee"
port_id             = "0x020200"
port_name           = "0x21000024ff21d2ee"
port_state          = "Online"
port_type           = "NPort (fabric via point-to-point)"
speed               = "8 Gbit"
supported_classes   = "Class 3"
supported_speeds    = "1 Gbit, 2 Gbit, 4 Gbit, 8 Gbit"
symbolic_name       = "QLE2562 FW:v5.03.02 DVR:v8.03.01.04.05.05-k"
system_hostname     = ""
tgtid_bind_type     = "wwpn (World Wide Port Name)"
uevent              = 

Device = "host5"
Device path = "/sys/devices/pci0000:00/0000:00:03.0/0000:15:00.0/host5"
ct                  =
edc                 = 
els                 =
fw_dump             =
nvram               = "ISP "
optrom_ctl          = 
optrom              =
reset               = 
sfp                 = ""
uevent              = 
vpd                 = "("


Class Device = "host6"
Class Device path = "/sys/class/fc_host/host6"
fabric_name         = "0x10000005331842a7"
issue_lip           = 
node_name           = "0x20000024ff21d2ef"
port_id             = "0x020200"
port_name           = "0x21000024ff21d2ef"
port_state          = "Online"
port_type           = "NPort (fabric via point-to-point)"
speed               = "8 Gbit"
supported_classes   = "Class 3"
supported_speeds    = "1 Gbit, 2 Gbit, 4 Gbit, 8 Gbit"
symbolic_name       = "QLE2562 FW:v5.03.02 DVR:v8.03.01.04.05.05-k"
system_hostname     = ""
tgtid_bind_type     = "wwpn (World Wide Port Name)"
uevent              = 

Device = "host6"
Device path = "/sys/devices/pci0000:00/0000:00:03.0/0000:15:00.1/host6"
ct                  =
edc                 = 
els                 =
fw_dump             =
nvram               = "ISP "
optrom_ctl          = 
optrom              =
reset               = 
sfp                 = ""
uevent              = 
vpd                 = "("


[root@isvx7 proc]#

Oracle Log Writer(LGWR) process on an commit

What I wanted to understand here was the behaviour of the Log Writer(LGWR) process when a "commit" is issued at the sqlplus prompt after inserting some date onto the table;

Here is the list of all the log files

SQL> select member from v$logfile;

MEMBER
--------------------------------------------------------------------------------
/u02/app/oracle/oradata/ABC/onlinelog/o1_mf_3_6kt8lo0w_.log
/u02/app/oracle/flash_recovery_area/ABC/onlinelog/o1_mf_3_6kt8lomz_.log
/u02/app/oracle/oradata/ABC/onlinelog/o1_mf_2_6kt8lms1_.log
/u02/app/oracle/flash_recovery_area/ABC/onlinelog/o1_mf_2_6kt8lnf2_.log
/u02/app/oracle/oradata/ABC/onlinelog/o1_mf_1_6kt8lkp2_.log
/u02/app/oracle/flash_recovery_area/ABC/onlinelog/o1_mf_1_6kt8lm49_.log

6 rows selected.

SQL>

lsof tells us the PID of the process that has opened the file, and also the FD (file descriptor) number.

[root@isvx3 ~]# lsof /u02/app/oracle/oradata/ABC/onlinelog/o1_mf_3_6kt8lo0w_.log
COMMAND   PID   USER   FD   TYPE DEVICE     SIZE     NODE NAME
oracle  32011 oracle   22u   REG    8,3 52429312 68747408 /u02/app/oracle/oradata/ABC/onlinelog/o1_mf_3_6kt8lo0w_.log
[root@isvx3 ~]# lsof /u02/app/oracle/flash_recovery_area/ABC/onlinelog/o1_mf_3_6kt8lomz_.log
COMMAND   PID   USER   FD   TYPE DEVICE     SIZE     NODE NAME
oracle  32011 oracle   23u   REG    8,3 52429312 68747409 /u02/app/oracle/flash_recovery_area/ABC/onlinelog/o1_mf_3_6kt8lomz_.log
[root@isvx3 ~]# lsof /u02/app/oracle/oradata/ABC/onlinelog/o1_mf_2_6kt8lms1_.log
COMMAND   PID   USER   FD   TYPE DEVICE     SIZE     NODE NAME
oracle  32011 oracle   20u   REG    8,3 52429312 68747406 /u02/app/oracle/oradata/ABC/onlinelog/o1_mf_2_6kt8lms1_.log
[root@isvx3 ~]# lsof /u02/app/oracle/flash_recovery_area/ABC/onlinelog/o1_mf_2_6kt8lnf2_.log
COMMAND   PID   USER   FD   TYPE DEVICE     SIZE     NODE NAME
oracle  32011 oracle   21u   REG    8,3 52429312 68747407 /u02/app/oracle/flash_recovery_area/ABC/onlinelog/o1_mf_2_6kt8lnf2_.log
[root@isvx3 ~]# lsof /u02/app/oracle/oradata/ABC/onlinelog/o1_mf_1_6kt8lkp2_.log
COMMAND   PID   USER   FD   TYPE DEVICE     SIZE     NODE NAME
oracle  32011 oracle   18u   REG    8,3 52429312 68747402 /u02/app/oracle/oradata/ABC/onlinelog/o1_mf_1_6kt8lkp2_.log
[root@isvx3 ~]# lsof /u02/app/oracle/flash_recovery_area/ABC/onlinelog/o1_mf_1_6kt8lm49_.log
COMMAND   PID   USER   FD   TYPE DEVICE     SIZE     NODE NAME
oracle  32011 oracle   19u   REG    8,3 52429312 68747405 /u02/app/oracle/flash_recovery_area/ABC/onlinelog/o1_mf_1_6kt8lm49_.log
[root@isvx3 ~]#

I checked to make sure 32011 was indeed the log writer process.

[root@isvx3 ~]# ps -ef | grep lgwr
oracle    9435     1  0  2010 ?        00:00:00 asm_lgwr_+ASM
root     13979 13779  0 17:20 pts/3    00:00:00 grep lgwr
oracle   32011     1  0  2010 ?        00:00:00 ora_lgwr_abc

Then I started stracing the log writer process and redirecting the output to a file /tmp/lgwr.txt

[root@isvx3 ~]# strace -o /tmp/lgwr.txt -p 32011
Process 32011 attached - interrupt to quit

Next I did an intert into my movies table to see if that causes the log writer process to do anything.

 SQL> insert into movies values (12, 'Masoon', 'naseeruddin Shah', 'Shabana Azmi',1983);

1 row created.

SQL> commit;

Commit complete.

On another window tailed the output from the lgwr

[root@isvx3 ~]# tail -f /tmp/lgwr.txt 

I observed that there were pwrite() calls made by LGWR right after the commit statement was issued at the sql prompt.

Next, I updated the last inset to change 'naseeruddin Shah' to Naseeruddin Shah'

SQL> update movies set actor = 'Naseeruddin Shah' where actor = 'naseeruddin Shah';

1 row updated.

The following pwrites were seen in the strace output after the update statement

[root@isvx3 ~]# cat /tmp/lgwr.txt | grep pwrite
pwrite(24, "\1\"\0\0\177\17\0\0\22\0\0\0\20\200!\177<\2\0\0\5\0\0\0\344\375\22\0\1\0\6\0"..., 1024, 2031104) = 1024
pwrite(25, "\1\"\0\0\177\17\0\0\22\0\0\0\20\200!\177<\2\0\0\5\0\0\0\344\375\22\0\1\0\6\0"..., 1024, 2031104) = 1024

Then I went on to commit the change

SQL> commit;

Commit complete. On enter the commit, the following pwrite was made to the LGWR process

pwrite(24, "\1\"\0\0\201\17\0\0\22\0\0\0\20\200O\244\214\0\0\0\5\0\0\0\357\375\22\0\1\0\0\0"..., 512, 2032128) = 512
pwrite(25, "\1\"\0\0\201\17\0\0\22\0\0\0\20\200O\244\214\0\0\0\5\0\0\0\357\375\22\0\1\0\0\0"..., 512, 2032128) = 512