Oracle 11.2.0.3 SCAN and VIP IPs are not reachable from other subnets.

There is Linux bug in Oracle 11.2.0.3 concerning SCAN and VIP IPs. When you reboot node from RAC or there is failover where these IP's are moving between nodes then the ip address is
not pingable from a different subnet. This seems to be only Linux problem. And everything is working normally in same cluster and same subnet servers. Problem is that ARP table (on router or on firewall) is getting wrong MAC addresses for these IPs.

When this problem is on and you try to connect this database from server which is in other subnet you get:
ERROR:
ORA-12170: TNS:Connect timeout occurred

As a workaround you can run following:
 After vip failover, run command
   /sbin/arping -U -c 3 -I <public NIC for vip> <vip ip address>
to update the ARP table of router.
or you can also clean problematic IPs from ARP table on your router/firewall



There is bug report and fix for this problem in MOS (My Oracle Support):

Bug 13440962 - Different subnet failed to connect to vip after restart vip

[ID 13440962.8]

You can find the fix from MOS when you Search from the Patches & Updates with patch number: 13440962

When you are installing this patch you might get errors from file/directory permissions. I needed to give read/write permission for oracle user in following directories (and their files):
$GRID_HOME/lib
$GRID_HOME/jlib
$GRID_HOME/crs
$GRID_HOME/bin

And remember to check that under bin directory following right are set (this needs to be exactly like this. If these are not correct database won't start at all.):
chmod 6751 oracle
chmod 4750 jssu

and I also needed to create this directory (for oracle user):
$GRID_HOME/.patch_storage

This patch like every other patches should be tested first in test environment.