ORA-1654: unable to extend index TPCC.ORDERS_I1 by 128 in tablespace USERS

While running the Hammerora workload against the database that I had created. I noticed that though the workload was running, the "Transaction Counter" was still showing 0.

I looked into the alert log of the database instance, and notice lots of ORA-1654 being generated.

ORA-1654: unable to extend index TPCC.ORDERS_I1 by 128 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 8192 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 128 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 8192 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 128 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 8192 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 128 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 8192 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 128 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 8192 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 128 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 8192 in tablespace               USERS ORA-1654: unable to extend index TPCC.ORDERS_I1 by 128 in tablespace               USERS

To fix the issue I first found the datafile names and there sizes for the USERS tablespace.

SQL> select file_name,bytes/1024/1024 "Megs" from dba_data_files where tablespace_name=   2  'USERS';  SQL> select file_name,bytes/1024/1024 "Megs" from dba_data_files where tablespace_name='USERS'; FILE_NAME --------------------------------------------------------------------------------       Megs ---------- /oracle_mvfs/testorcl/users01.dbf           32735 /oracle_mvfs/testorcl/users02.dbf           30720 /oracle_mvfs/testorcl/users03.dbf           30720 .......... .......... ..........

Next, I increased the size of the datafiles as follows.

SQL> alter database datafile '/oracle_mvfs/testorcl/users20.dbf' resize 32735M; Database altered. SQL> alter database datafile '/oracle_mvfs/testorcl/users19.dbf' resize 32735M; Database altered.

Voila, we were back on track.

ORA-24550: signal received: [si_signo=11] [si_errno=0] [si_code=1] [si_int=-1066996032]

While trying to run the tpc-c workload using Hammerora tool I got the following ORA-24550 error message and the client GUI crashed.

The exact error message was:

ORA-24550: signal received: [si_signo=11] [si_errno=0] [si_code=1] [si_int=-1066996032] [si_ptr=0x2b8bc066eec0] [si_addr=0xef000a72bd] kpedbg_dmp_stack()+314<-kpeDbgCrash()+166<-kpeDbgSignalHandler()+158<-skgesig_sigactionHandler()+217<-__restore_rt()<-DisplayText()+621

After trying out a few things, I tried to connect to the database(testorcl) using the hammer ora users(tpcc) account and got the error message ORA-12541: TNS:no listener

[oracle@isvx7 ~]$ sqlplus tpcc/xxxx@testorcl SQL*Plus: Release 11.2.0.1.0 Production on Thu Jun 9 11:23:08 2011 Copyright (c) 1982, 2009, Oracle.  All rights reserved. ERROR: ORA-12541: TNS:no listener

I started the listener using lsnrctl, and then started the Hammerora client againg to generate the TPC-C workload, this time things worked fine.

vxlicinst ERROR V-21-1-19 Duplicate License key detected

I was working on Symantec SmartTier and multi-volumed file systems when I got the message that my license has expired and that I need a new one.

[root@isvx7 ~]# fsppadm query /oracle_mvfs UX:vxfs fsppadm: ERROR: V-3-26550: The command failed with message - à< UX:vxfs fsppadm: ERROR: V-3-26550: The command failed with message - /sbin/vxlictest -n "VERITAS File System" -sup -f "Quality of Storage Service" UX:vxfs fsppadm: ERROR: V-3-26540: Valid license is not found

I got a new license, and tried to add it when I got the message "Duplicate License key detected"

[root@isvx7 ~]# cd /sbin [root@isvx7 sbin]# ls vxlic* vxlicinst  vxlicrep  vxlictest [root@isvx7 sbin]# /sbin/vxlicinst Symantec License Manager vxlicinst utility version 3.02.51.010 Copyright (C) 1996-2010 Symantec Corporation. All rights reserved. Enter your license key : MMMM-NNNN-OOOO-PPPP-QQQQ-RRRR-SSSS-T12 vxlicinst ERROR V-21-1-19 Duplicate License key detected [root@isvx7 sbin]#

The workaround for this is move all the files under /etc/vx/license/lic to some other directory, and then try to apply the license again.

[root@isvx7 sbin]# cd /etc/vx/licenses/lic [root@isvx7 lic]# ls AAAA-BBB-CCCC-DDDD-EEEE-FFFF-GGGG-HHHH-III.vxlic JJJJ-KKKK-LLLL-MMMM-NNNN-OOOO-PPPP-QQQQ-RRR.vxlic SSSS-TTTT-UUUU-VVVV-WWWW-XXXX-YYYY-ZZZZ-AAA.vxlic BBBB-CCCC-DDDD-EEEE-FFFF-GGGG-HHHH-III.vxlic [root@isvx7 lic]# mv *.* /root/vxlicenses [root@isvx7 ~]# /sbin/vxlicinst Symantec License Manager vxlicinst utility version 3.02.51.010 Copyright (C) 1996-2010 Symantec Corporation. All rights reserved. Enter your license key : MMMM-NNNN-OOOO-PPPP-QQQQ-RRRR-SSSS-T12 Number of days left for Demo = 364 License key successfully installed for VERITAS Storage Foundation Enterprise HA License key successfully installed for VERITAS Volume Manager License key successfully installed for VERITAS Cluster Server License key successfully installed for VERITAS File System [root@isvx7 ~]#

ORA-00265: instance recovery required, cannot set ARCHIVELOG mode

I wanted to turn archiving for my database. To turn on archive log mode the database must be mounted, but not open. If the database is already open, first shutdown the database using "shutdown" or "shutdown immediate". If you use "shutdown abort" to shutdown the database, then while running the SQL> alter database archivelog; you will get the ORA-00265: instance recovery required, cannot set ARCHIVELOG mode

The easiest way to get around this would be to start the database again using startup, and the shutting it again using "shutdown" or "shutdown immediate". This will do the trick.

-bash-3.2$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.1.0 Production on Wed Jun 8 12:06:23 2011 Copyright (c) 1982, 2009, Oracle.  All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> archive log list; Database log mode              No Archive Mode Automatic archival             Disabled Archive destination            /u01/app/oracle/product/11.2.0/dbhome_1/dbs/arch Oldest online log sequence     132361 Current log sequence           132363 SQL> alter database archivelog; alter database archivelog * ERROR at line 1: ORA-01126: database must be mounted in this instance and not open in any instance SQL> shutdown abort; ORACLE instance shut down. SQL> startup mount; ORACLE instance started. Total System Global Area 2.6991E+10 bytes Fixed Size                  2213976 bytes Variable Size            1.7985E+10 bytes Database Buffers         8858370048 bytes Redo Buffers              145174528 bytes Database mounted. SQL> alter database archivelog; alter database archivelog * ERROR at line 1: ORA-00265: instance recovery required, cannot set ARCHIVELOG mode SQL> shutdown; ORA-01109: database not open Database dismounted. ORACLE instance shut down. SQL> SQL> startup mount; ORACLE instance started. Total System Global Area 2.6991E+10 bytes Fixed Size                  2213976 bytes Variable Size            1.7985E+10 bytes Database Buffers         8858370048 bytes Redo Buffers              145174528 bytes Database mounted. SQL> alter database archivelog; alter database archivelog * ERROR at line 1: ORA-00265: instance recovery required, cannot set ARCHIVELOG mode SQL> shutdown; ORA-01109: database not open Database dismounted. ORACLE instance shut down. SQL> startup; ORACLE instance started. Total System Global Area 2.6991E+10 bytes Fixed Size                  2213976 bytes Variable Size            1.7985E+10 bytes Database Buffers         8858370048 bytes Redo Buffers              145174528 bytes Database mounted. Database opened. SQL> shutdown immediate; Database closed. Database dismounted. ORACLE instance shut down. SQL> startup mount; ORACLE instance started. Total System Global Area 2.6991E+10 bytes Fixed Size                  2213976 bytes Variable Size            1.7985E+10 bytes Database Buffers         8858370048 bytes Redo Buffers              145174528 bytes Database mounted. SQL> alter database archivelog; Database altered. SQL> SQL>  archive log list; Database log mode              Archive Mode Automatic archival             Enabled Archive destination            /u01/app/oracle/product/11.2.0/dbhome_1/dbs/arch Oldest online log sequence     132362 Next log sequence to archive   132364 Current log sequence           132364 SQL>

mount: 0506-324 Cannot mount /dev/oradata_lv on /oradata: There is a request to a device or address that does not exist.

I have a two node cluster sharing a Storwize V7000 between then. Volume groups "oradata_vg" and "oralogs_vg" were created on the shared LUNs.

I then mounted file systesms /ordata and /oralogs on the volume groups. I was also able to manualy switchover the file system between the two nodes of the cluster.

But now when I tried to mount /oradata on the first node I got the "mount: 0506-324" error message.

isvp14_ora> mount /oradata mount: 0506-324 Cannot mount /dev/oradata_lv on /oradata: There is a request to a device or address that does not exist.

On doing an lspv on the node I noticed that the volume group was not active. I did an varyonvg on that volume, but that failed.

isvp14_ora> lspv hdisk0          00f62a6c98758859                    rootvg          active hdisk1          00f62a6ca2279d3e                    swap            active hdisk2          00f62a6cdf8874f5                    None hdisk3          00f62a6cdf8875ff                    None hdisk4          00f62a6ce57d00d4                    oradata_vg hdisk5          00f62a6bdf88762c                    None hdisk6          00f62a6bdf887797                    oralogs_vg hdisk7          00f62a6bdf887802                    None

I noticed that the volume group was active on the second node, so I did a varoffvg of the volume groups. I then did a varyonvg of the volume groups on the first node. I was then able to mount the file systems

isvp15_ora> lspv hdisk0          00f62a6c98758859                    rootvg          active hdisk1          00f62a6ca2279d3e                    swap            active hdisk2          00f62a6cdf8874f5                    None hdisk3          00f62a6cdf8875ff                    None hdisk4          00f62a6ce57d00d4                    oradata_vg      active hdisk5          00f62a6bdf88762c                    None hdisk6          00f62a6bdf887797                    oralogs_vg      active hdisk7          00f62a6bdf887802                    None isvp15_ora> varyoffvg oradata_vg isvp15_ora> varyoffvg oralogs_vg isvp14_ora> varyonvg oradata_vg isvp14_ora> varyonvg oralogs_vg isvp14_ora> mount /oradata

Manually starting and stopping Oracle 11gR2 from the command line, and a shell script

A colleague of mine asked me how to start Oracle database from the command like without getting into SQL.

To start and stop the database, testora is the name of our database

bash-3.2$ srvctl start database -d testora bash-3.2$ srvctl stop database -d testora

To shutdown the ASM instance

bash-3.2$ srvctl stop asm -f

Start the ASM instance and the diskgroups:

bash-3.2$ srvctl start asm

Script to startup Oracle(startup.ksh)

#!/usr/bin/ksh
SQLDBA="sqlplus /nolog"
$SQLDBA <<EOF
connect / as sysdba
startup
quit
EOF

Script to shutdown Oracle(shutdown.ksh)

#!/usr/bin/ksh
SQLDBA="sqlplus /nolog"
$SQLDBA <<EOF
connect / as sysdba
shutdown abort
quit
EOF 

Script to startup Oracle ASM(startup_asm.ksh)

#!/usr/bin/ksh
export ORACLE_SID=+ASM
SQLDBA="sqlplus /nolog"

$SQLDBA <<EOF
connect / as sysdba
connect sys/test123 as sysasm
startup
quit
EOF
 

Script to shutdown Oracle ASM(shutdown_asm.ksh)

#!/usr/bin/ksh
ORACLE_SID="+ASM"
export ORACLE_SID
SQLDBA="sqlplus / as sysdba"

$SQLDBA <<EOF
connect sys as sysdba
connect sys/test123 as sysasm
shutdown abort
quit
EOF

Got wget, bash, unzip for AIX 7.1?

Well I was looking around for all these utilities for a while untill i came across this page:
ftp://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/README.txt

isvp14_ora> ftp ftp.software.ibm.com       Name> ftp       Password> your e-mail address           ftp> cd aix/freeSoftware/aixtoolbox/RPMS/ppc/wget           ftp> binary           ftp> get wget-1.9.1-1.aix5.1.ppc.rpm           ftp> quit       isvp14_ora> rpm -hUv wget-1.9.1-1.aix5.1.ppc.rpm       isvp14_ora> wget -r -nd ftp://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/ezinstall/ppc

You will now have the following files in the directory that you created:

isvp14_ora> ls getapp-dev.sh       getgnome.base.sh    getkde3.all.sh Xsession.kde        getbase.sh          getkde2.all.sh      getkde3.base.sh Xsession.kde2       getdesktop.base.sh  getkde2.base.sh     getkde3.opt.sh getgnome.apps.sh    getkde2.opt.sh      isvp14_ora> chmod +x get*.sh

Run the script getbase.sh this will create a directory called base, ftp the rpm's into it.

isvp14_ora> isvp14_ora> cd base isvp14_ora> ls bash-3.2-1.aix5.2.ppc.rpm          rpm-3.0.5-52.aix5.3.ppc.rpm bzip2-1.0.5-3.aix5.3.ppc.rpm       rpm-build-3.0.5-52.aix5.3.ppc.rpm gettext-0.10.40-8.aix5.2.ppc.rpm   rpm-devel-3.0.5-52.aix5.3.ppc.rpm gzip-1.2.4a-10.aix5.2.ppc.rpm      tar-1.14-2.aix5.1.ppc.rpm info-4.6-1.aix5.1.ppc.rpm          unzip-5.51-1.aix5.1.ppc.rpm patch-2.5.4-4.aix4.3.ppc.rpm       popt-1.7-2.aix5.1.ppc.rpm isvp14_ora>

Install the rpms that you need:

isvp14_ora> rpm -hUv unzip-5.51-1.aix5.1.ppc.rpm isvp14_ora> rpm -hUv zip-2.3-3.aix4.3.ppc.rpm isvp14_ora> rpm -hUv bash-3.2-1.aix5.2.ppc.rpm There we go, we now have bash with AIX 7.1 isvp14_ora> bash bash-3.2#