Wednesday, March 28, 2012

Increasing the jfs2 file system size on AIX beyond 64GB

In my earlier post I talk about increasin the size of a file system on AIX

http://shettymayur.blogspot.com/2011/01/how-to-increase-file-system-size-on-aix.html

In this post I created a 100GB SAN volume, which I mapped to my AIX 7.1 host.

Using smitty I created a 64GB jfs2 file system on the volume.


/dev/fslv02       64.00     52.61   18%     5459     1% /bigfs
isvp14_ora> lslv fslv02
LOGICAL VOLUME:     fslv02                 VOLUME GROUP:   bigfs
LV IDENTIFIER:      00f62a6b00004c0000000136568e8fcf.2 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       opened/syncd
TYPE:               jfs2                   WRITE VERIFY:   off
MAX LPs:            512                    PP SIZE:        128 megabyte(s)
COPIES:             1                      SCHED POLICY:   parallel
LPs:                512                    PPs:            512
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        /bigfs                 LABEL:          /bigfs
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO
isvp14_ora>

The volume that was mapped to the host was 100GB, but when I tried to increase the file system size beyond 64GB I get the below message.

isvp14_ora> chfs -a size=90G /bigfs
0516-787 extendlv: Maximum allocation for logical volume fslv02 is 512.

We see that it's complaining that the maximum number LP is 512. LV can go up to 512 LPs, each LP is 128 MB , so  512 x 128MB= 64 GB

Below we below see the command to increase the LPs to 1024, this will give us a max file system of 128 GB


isvp14_ora> chlv -x'1024' fslv02
isvp14_ora> lslv fslv02
LOGICAL VOLUME:     fslv02                 VOLUME GROUP:   bigfs
LV IDENTIFIER:      00f62a6b00004c0000000136568e8fcf.2 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       opened/syncd
TYPE:               jfs2                   WRITE VERIFY:   off
MAX LPs:            1024                   PP SIZE:        128 megabyte(s)
COPIES:             1                      SCHED POLICY:   parallel
LPs:                512                    PPs:            512
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        /bigfs                 LABEL:          /bigfs
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO

isvp14_ora> chfs -a size=90G /bigfs
Filesystem size changed to 188743680
isvp14_ora> df -g
/dev/fslv02       90.00     78.60   13%     5459     1% /bigfs

Tuesday, March 6, 2012

Installing Sqoop on Red Hat 4.1.2-50

As I wanted to install sqoop on my Red Hat box, I went to https://ccp.cloudera.com/display/CDHDOC/Sqoop+Installation where I found some install steps. For Red Hat Linux it said to do a "yum install", but that failed on my box.

[root@isvx7 ~]# sudo yum install sqoop
Loaded plugins: rhnplugin, security
There was an error communicating with RHN.
RHN Satellite or RHN Classic support will be disabled.
Error communicating with server. The message was:

Error Message:
        Please run rhn_register as root on this client
Error Class Code: 9
Error Class Info: Invalid System Credentials.
Explanation:
     An error has occurred while processing your request. If this problem
     persists please enter a bug report at bugzilla.redhat.com.
     If you choose to submit the bug report, please be sure to include
     details of what you were trying to do when this error occurred and
     details on how to reproduce this problem.

Setting up Install Process
No package sqoop available.
Nothing to do


I then ended up installing the mapr packages from http://package.mapr.com/releases/v1.2.2/redhat/

As per the http://www.mapr.com/doc/display/MapR/Local+Packages+-+Red+Hat you required the following packages

RolePackage Dependencies
Sqoopmapr-sqoop-1.2.2.12455.GA-1.x86_64.rpm mapr-core-1.2.2.12455.GA-1.x86_64.rpm
mapr-hbase-internal-1.2.2.12455.GA-1.x86_64.rpm
mapr-sqoop-internal-1.2.2.12455.GA-1.x86_64.rpm


[root@isvx7 bigdata]# wget http://package.mapr.com/releases/v1.2.2/redhat/mapr-s            qoop-internal-1.2.2.12455.GA.v1.3.0-1.x86_64.rpm
--2012-03-06 15:52:34--  http://package.mapr.com/releases/v1.2.2/redhat/mapr-sqo            op-internal-1.2.2.12455.GA.v1.3.0-1.x86_64.rpm
Resolving package.mapr.com... 205.251.215.104, 205.251.215.186, 205.251.215.219,             ...
Connecting to package.mapr.com|205.251.215.104|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3425149 (3.3M) [application/x-redhat-package-manager]
Saving to: `mapr-sqoop-internal-1.2.2.12455.GA.v1.3.0-1.x86_64.rpm'

100%[======================================>] 3,425,149    347K/s   in 8.4s

2012-03-06 15:52:42 (399 KB/s) - `mapr-sqoop-internal-1.2.2.12455.GA.v1.3.0-1.x8            6_64.rpm' saved [3425149/3425149]


[root@isvx7 bigdata]# wget http://package.mapr.com/releases/v1.2.2/redhat/mapr-sqoop-1.2.2.12455.GA-1.x86_64.rpm
--2012-03-06 15:48:30--  http://package.mapr.com/releases/v1.2.2/redhat/mapr-sqoop-1.2.2.12455.GA-1.x86_64.rpm
Resolving package.mapr.com... 216.137.43.10, 216.137.43.125, 216.137.43.136, ...
Connecting to package.mapr.com|216.137.43.10|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1973 (1.9K) [application/x-redhat-package-manager]
Saving to: `mapr-sqoop-1.2.2.12455.GA-1.x86_64.rpm'

100%[======================================>] 1,973       --.-K/s   in 0s


2012-03-06 15:48:30 (125 MB/s) - `mapr-sqoop-1.2.2.12455.GA-1.x86_64.rpm' saved   [1973/1973]




[root@isvx7 bigdata]# wget http://package.mapr.com/releases/v1.2.2/redhat/mapr-c  ore-1.2.2.12455.GA-1.x86_64.rpm
--2012-03-06 15:48:49--  http://package.mapr.com/releases/v1.2.2/redhat/mapr-cor  e-1.2.2.12455.GA-1.x86_64.rpm
Resolving package.mapr.com... 216.137.43.136, 216.137.43.174, 216.137.43.203, ..  .
Connecting to package.mapr.com|216.137.43.136|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 112941895 (108M) [application/x-redhat-package-manager]
Saving to: `mapr-core-1.2.2.12455.GA-1.x86_64.rpm'

100%[======================================>] 112,941,895  678K/s   in 3m 36s

2012-03-06 15:52:25 (511 KB/s) - `mapr-core-1.2.2.12455.GA-1.x86_64.rpm' saved [  112941895/112941895]


[root@isvx7 bigdata]# ls
mapr-core-1.2.2.12455.GA-1.x86_64.rpm
mapr-sqoop-1.2.2.12455.GA-1.x86_64.rpm
mapr-sqoop-internal-1.2.2.12455.GA.v1.3.0-1.x86_64.rpm



[root@isvx7 bigdata]# rpm -i mapr-core-1.2.2.12455.GA-1.x86_64.rpm
[root@isvx7 bigdata]# rpm -i mapr-sqoop-internal-1.2.2.12455.GA.v1.3.0-1.x86_64.            rpm
[root@isvx7 bigdata]# rpm -i mapr-sqoop-1.2.2.12455.GA-1.x86_64.rpm
[root@isvx7 bigdata]# which sqoop
/usr/bin/sqoop

At this point only the sqoop rpm's were added, the hbase rpm had not yet been added to the linux box that is why we get a warning message when invoking sqoop as shown below.

 [root@isvx7 bigdata]# sqoop -h
Warning: /opt/mapr/hbase/hbase-0.90.4 does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
No such sqoop tool: -h. See 'sqoop help'.

We now get the hbase rpm and then install the hbase rpm on the linux host.

[root@isvx7 bigdata]# wget http://package.mapr.com/releases/v1.2.2/redhat/mapr-hbase-internal-1.2.2.12455.GA.v0.90.4-1.x86_64.rpm
--2012-03-06 16:47:39--  http://package.mapr.com/releases/v1.2.2/redhat/mapr-hbase-internal-1.2.2.12455.GA.v0.90.4-1.x86_64.rpm
Resolving package.mapr.com... 205.251.215.160, 205.251.215.180, 205.251.215.222, ...
Connecting to package.mapr.com|205.251.215.160|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 29024196 (28M) [application/x-redhat-package-manager]
Saving to: `mapr-hbase-internal-1.2.2.12455.GA.v0.90.4-1.x86_64.rpm'

100%[========================================>] 29,024,196  1.56M/s   in 38s

2012-03-06 16:48:17 (755 KB/s) - `mapr-hbase-internal-1.2.2.12455.GA.v0.90.4-1.x86_64.rpm' saved [29024196/29024196]

[root@isvx7 bigdata]#

[root@isvx7 bigdata]# rpm -i mapr-hbase-internal-1.2.2.12455.GA.v0.90.4-1.x86_64.rpm


No warning messages this time around.

[root@isvx7 bigdata]# sqoop help
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See 'sqoop help COMMAND' for information on a specific command.
[root@isvx7 bigdata]# sqoop version
Sqoop 1.3.0
git commit id
Compiled by sathya@satdev1 on Tue Aug 23 18:03:37 PDT 2011
[root@isvx7 bigdata]#