Slow connection between the Oracle RAC nodes

While setting up our Oracle RAC hardware running AIX 7.1 we noticed that a simple 'rsh' between the nodes would take really long. In this case we were trying to see if we could rsh from one node to the other node using the private interconnect which were using 10 GigE NICs.

As you see here that something like "rsh 192.168.150.22 'date'" took an awful long time.

bash-4.2# time rsh 192.168.150.22 'date' Thu Apr 11 18:30:41 CDT 2013 real    1m11.372s user    0m0.002s sys     0m0.002s bash-4.2#

This thing was that we did not have a 'hosts' entry in the /etc/netsvc.conf file. The /etc/netsvc.conf file is used to specify the ordering of name resolution. The order we specify in the /etc/netsvc.conf will override the default ordering, and use the order specified by us for the 'hosts' keyword.

After we added the following line to the /etc/netsvc.conf, everything worked as expected.
hosts=local,bind

Here is the rsh after adding the hosts field to the /etc/netsvc.conf

bash-4.2# time rsh 192.168.150.22 'date' Thu Apr 11 18:31:58 CDT 2013 real    0m0.107s user    0m0.002s sys     0m0.002s bash-4.2#