Oracle ORA-7445 [kkzufst] errors in alertlog.

ORA-7445 [kkzufst] errors in alertlog are result from bugs in materialized views refresh jobs.

If you got these bugs you'll see these kind of errors in your database alertlog
(there is always ORA-07445 and kkzufst in error but other clauses can vary.):
--
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x1000] [PC:0x44BBFCE, kkzufst()+111] [flags: 0x0, count: 1]
Errors in file...
ORA-07445 : exception encountered: core dump [kkzufst()+111] ...
Incident details in:...
--

In incident log you'll see something like this (here you can also see problematic SQL or PL/SQL clause in "Current SQL Statement ..." section)
(there is always ORA-07445 and kkzufst in error but other clauses can vary.):
--
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x1000] [PC:0x44BBFCE, kkzufst()+111] [flags: 0x0, count: 1]
.
.
.
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x3, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=...

--


Fixes:
Bugs are more accurately 9554075 and 9656434
These both bugs are fixed in Oracle version 11.2.0.2 SPS (Server Patch Set).
And in Oracle version 12 fixes are in following versions: 9554075 -> version Oracle 12.1.0.1 (Base Release). 9656434 -> version Oracle 12.2 (Future release. (Not yet released))


More info about these bugs can be find from MOS (My Oracle Support) document:  
1288415.1

Oracle 12c enhancements for SCAN

There are new features for SCAN in Oracle 12c:

1. SCAN and Oracle Clusterware managed VIPs now support IPv6 based IP addresses.
So now we can use also IPv6 addresses with SCAN. If you check SCAN configs with:
srvctl config scan
it will list you more info: SCAN name, IPv4 address, IPv6 address, SCAN number+VIP



2. SCAN is by default restricted to only accept service registration from nodes in the cluster.
If you want to add nodes (for example for the database that are not using cluster's private interconnect) then you can do it with command:
srvctl modify scan_listener -invitednodes <new_database_name> –update
You can also run this modify before you present real node in the cluster.
 
If you check SCAN-listener configs with:
srvctl config scan_listener
it will list you more info: SCAN-listener name+port, Registration invited nodes, Registration invited subnets



3. SCAN supports multiple subnets in the cluster (one SCAN per subnet).

Before you can do this you must enable multiple subnets for the cluster (This is a post-installation task an cannot be made in Grid Infra installation). 

 


NOTE! Here you can find more info about SCAN and Oracle 12c enhancements :
http://www.oracle.com/technetwork/products/clustering/overview/scan-129069.pdf

Oracle 12c New Features for table Defaults

With Oracle 12c there is improvements in tables default values.

Starting on Oracle 12c you can use sequence nextval straight in column default or you can use "generated as identity" for column default (this will generate new sequence for table and start use it for column default values) This new sequence is linked with table so if you remove the table also the sequence is removed. You can also use "generated by default as identity" when you create new table sequence. This will allow you to override sequence value in inserts. If you use default word as column value or leave column and it's value out of the insert the sequence is used but you can also give new value for this column when "by default" is used.

With Oracle 12c you can also give default values for null columns. So if you are inserting null value into column that have for example this default settings: "default ON NULL 11" then 11 value will be inserted into table instead of null.

Examples:

Create table clause with using existing sequence:
SQL> CREATE TABLE TEST_TABLE ( id int seq_test_id.nextval primary key, name varchar2(32) );


Create table clause with generating new identity sequence:
SQL> CREATE TABLE TEST_TABLE ( id int generated as identity primary key, name varchar2(32) );

Create table clause with generating new identity sequence with sequence init values:
SQL> CREATE TABLE TEST_TABLE( id int generated as identity (start with 10000) primary key, name varchar2(32) );

Override sequence value when table squence is created with "generated by default as identity":
SQL> INSERT INTO TEST_TABLE (id, name) VALUES ( 10, 'test_name');

Using sequence value when table squence is created with "generated by default as identity":
SQL> INSERT INTO TEST_TABLE (id, name) VALUES ( default, 'test_name');
or
SQL> INSERT INTO TEST_TABLE (name) VALUES ( 'test_name');

Oracle 11.2.0.3 ORA-04030: out of process memory when trying to allocate xxxx bytes (kxs-heap-w,KGL Iterator information)

If you get the ORA-04030: ... (kxs-heap-w,KGL Iterator information) error in alert log
and if you see obsoleted parent cursors lying around.
Then you are hitting Bug 12791981 - ORA-4030 due to "KGL Iterator information" . 
Look more from MOS (My Oracle Support) document 12791981.8 .

If you are trying to check those obsolete cursors you are probably hitting the same error ORA-04030 and then you know that there are also those obsoleted cursors in the database.

With this SQL you can check those obsoleted cursors:
select sql_id, address, IS_OBSOLETE, count(*) from v$sql where IS_OBSOLETE='Y' group by sql_id, address, IS_OBSOLETE having count(*)>=10;

Or just:
select count(*) from v$sql where IS_OBSOLETE='Y'; 

You can fix this bug with patch 12791981 .


NOTE! This bug can also be fixed with patch 14799269 (this is more complete fix according to MOS) but it is only available for 11.2.0.3.1 version and newer ones.

Oracle RAC cluster NIC bonding.

You need to do following steps to start using NIC bonding with Oracle RAC (this add both public and private interconnect bond interfaces):

NOTE! This operation needs full downtime from cluster databases. And depending your environment you might need server reboots during this settings.

1. Add new bond0 and bond1 interfaces for RAC cluster globally via oifcfg. public interface is bond0 and cluster_interconnect interface bond1:

Get current network interface configuration being used by cluster as oracle user:
oifcfg getif




Set new bond interfaces – Updates OCR (-global make these changes to all nodes on cluster) run these in one node as oracle user (IPs can be same as before but interface name is changing):
oifcfg setif -global bond0/10.77.5.0:public
oifcfg setif -global bond1/10.37.24.0:cluster_interconnect
oifcfg getif

2. Stop databases and disable + stop crs on all nodes:

Do this on one node as oracle user (and all databases in cluster):
srvctl status database -d <database_name>  
srvctl stop database -d <database_name>
srvctl status database -d <database_name>  

Do these on both nodes as root user:
crsctl disable crs
crsctl check crs
crstcl stop crs
crsctl check crs
 

3.Change OS network interfaces to use NIC bonding like this (this guide only add bond0 but it is better to make bond0 (eth0 and eth1) for public and bond1 (eth2 and eth3 ) for interconnect. And you don't need to make alias interfaces.): 

http://www.oracle-base.com/articles/linux/nic-channel-bonding.php



If you need to reboot nodes because of your environment then do it now.

4. Enable and start crs on all nodes:

Do these on both nodes as root user:
crsctl enable crs
crsctl check crs
crstcl start crs
crsctl check crs
 check that crs started on both nodes run as root user on one node:
crsctl stat res -t

 

5. Remove old interfaces from RAC cluster via oifcfg:

Get current pvt. interconnect info as oracle user in one node:
oifcfg getif
 
Delete old interfaces eth0 and eth1 as oracle user in one node:
oifcfg delif -global eth0/10.77.5.0
oifcfg delif -global eth1/10.37.24.0

Check that only new bond interfaces are visible as oracle user in one node:
oifcfg getif

6. Repair scan- and vip- addresses using bond0 instead of eth0 :

Check cluster current network status as oracle user:
srvctl status nodeapps
Check current VIP settings. run this on all nodes as oracle user:
srvctl config vip -n <node_name>

Check current SCAN settings. run this on one node as oracle user:
srvctl config scan

Stop cluster nodeapps as oracle user:
srvctl stop nodeapps

Change SCAN/VIP settings. run this on all nodes as root user (set correct IP for all nodes):
srvctl modify nodeapps -n <node_name> -A 10.77.5.129/255.255.255.0/bond0

Check current VIP settings. run this on all nodes as oracle user:
srvctl config vip -n <node_name>

Start cluster nodeapps as oracle user:
srvctl stop nodeapps
Check cluster current network status as oracle user:
srvctl status nodeapps

Check current SCAN settings. run this on one node as oracle user:
srvctl config scan


 

7. Restart crs to see that it is starting correctly:

Do these on both nodes as root user: 

crsctl check crs
crstcl stop crs
crsctl check crs
crstcl start crs
crsctl check crs

8. Restart databases:

Do this on one node as oracle user (and all databases in cluster):
srvctl status database -d <database_name> 
srvctl start database -d <database_name> 
srvctl status database -d <database_name>

9. Test that you can connect into databases via all SCAN/VIP IPs. You can do this for example via sqlplus.

 

Oracle 11.2.0.3.0 DATABASE CRASHED DUE TO ORA-240 AND ORA-15064

There is a bug in Oracle 11.2.0.3.0 which can make your database instance restarting itself.
If you get following errors in database alert.log you know this bug is affecting your database:
ORA-00240: control file enqueue held for more than 120 seconds
ORA-29770: global enqueue process LCK0 (OSID 12329) is hung for more than 70 seconds
ORA-15064: communication failure with ASM instance

There is bugfix for this problem and you can download it from My Oracle Support (MOS) patch:
13914613

Other way to fix this is to update your database to newest version where this is also fixed

More info about this can be find from MOS documents:
Database Instance Crashes with ORA-15064 ORA-03135 ORA-00240 on 11.2 (Doc ID 1487108.1)

Bug 13914613 - Excessive time holding shared pool latch in kghfrunp with auto memory management (Doc ID 13914613.8) 


 

Oracle ORA-21561 : OID generation failed

If Oracle client connection is giving "ORA-21561 : OID generation failed" error.
Like this:


sqlplus <test_user>/<test_user_password>@<database_name>

.
.
.

ERROR:
ORA-21561 : OID generation failed

Enter user-name: <username>/<password> @ <tns connect string>


Then the problem is most likely in the client machine hosts file.
Check that there is client machine fully qualified name and short name in the client machine hosts file. If these are missing you'll get ORA-21561 errors when trying to connect server.