Oracle CRS is not starting "has a disk HB, but no network HB, DHB has rcfg..." in ocssd log

Network problems in interconnect network or problems with interconnect interface can prevent CRS for starting.

If you look CRS check you'll see following:
[root@<node2> <node2>]# /u01/app/11.2.0/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager


CRS alert log shows following:
(/u01/app/11.2.0/grid/log/<node2>/alert<node2>.log):
.
.
.
2013-06-03 06:56:50.778
[/u01/app/11.2.0/grid/bin/cssdagent(13124)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'.
Details at (:CRSAGF00113:) {0:28:4} in /u01/app/11.2.0/grid/log/<node2>/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2013-06-03 06:56:50.779
[cssd(13138)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/<node2>/cssd/ocssd.log
.
.
.


ocssd log shows following:
 (/u01/app/11.2.0/grid/log/<node2>/cssd/ocssd.log) (this is complaining about node1 interconnect) :


2013-06-03 06:56:50.814: [    CSSD][3190012224]clssnmvDHBValidateNCopy: node 1, <node1>, has a disk HB, but no network HB, DHB has rcfg 216823918, wrtcnt, 48655493,
LATS 5387994, lastSeqNo 48655492, uniqueness 1365009957, timestamp 1370231810/927159338

2013-06-03 06:56:51.822: [    CSSD][3190012224]clssnmvDHBValidateNCopy: node 1, <node1>, has a disk HB, but no network HB, DHB has rcfg 216823918, wrtcnt, 48655494,
LATS 5389004, lastSeqNo 48655493, uniqueness 1365009957, timestamp 1370231811/927160338

2013-06-03 06:56:52.862: [    CSSD][3190012224]clssnmvDHBValidateNCopy: node 1, <node1>, has a disk HB, but no network HB, DHB has rcfg 216823918, wrtcnt, 48655495,
LATS 5390044, lastSeqNo 48655494, uniqueness 1365009957, timestamp 1370231812/927161338
.


You can check Ping and SSH between nodes via interconnect interface.
If they are not working then there is problem in network connection between cluster nodes. Fix the problem and CRS will start correctly. 

But if Ping and SSH did work between nodes via interconnect interface and still ocssd log did complain about interconnect HeartBeat (no network HB) then interconnect interface is jammed. You can try to restart it to get it fixed (NOTE! It is usually the working node interconnect interface that is needed to restart (like error message is saying in ocssd.log (it is complaining node1)). For example if node2 CRS is not starting then restart node1 interconnect interface ) :
[root@<node1> <node1>]# ifdown eth1
[root@<node1> <node1>]# ifup eth1
And check that eth1 is looking ok:
[root@<node1> <node1>]# ifconfig


After interface restart or network problem fix check that <node2> clusterware is starting again:
[root@<node2> <node2>]# /u01/app/11.2.0/grid/bin/crsctl check crs

NOTE: If clusterware is trying to connect long enough via interconnect without success it will give this message in its alert log:
[ohasd(7773)]CRS-2771:Maximum restart attempts reached for resource 'ora.cssd'; will not restart.

If this error occurs then you need to kill CRS processes manually or reboot <node2> to get it trying again the clusterware start (cssd start).

Oracle cannot allocate new log "Private strand flush not complete" or "Checkpoint not complete" in alert log.

It's not unusual to get "Private strand flush not complete" or "Checkpoint not complete" messages with cannot allocate new log in alert log. Both are relating redo log writing and they are not errors. But if you get these a lot then it is better to try to fix them.

1. Usually easiest way to fix these is add more Redo Logs and Redo Log groups. 

2. Other way is increase log files size.

3. Increasing the value for db_writer_processes (and use ASYNC I/O with Redo Logs ) can help with  these. These are increasing Redo writing speed.

4. You can also try to tune checkpoint to fix "Checkpoint not complete". Here you can find parameters to tune: Checkpoint Tuning and Troubleshooting Guide [ID 147468.1]

5. And also moving log files to faster disk is possible fix.


If you want more detailed info about these alert log messages read following MY Oracle Support (MOS) documents:

Alert Log Messages: Private Strand Flush Not Complete [ID 372557.1]

Can Not Allocate Log [ID 1265962.1]
Checkpoint Tuning and Troubleshooting Guide [ID 147468.1]

Oracle sequence last number check and creation clauses.

Sometimes for example when you are updating database you might need to check that sequences last used value does not change. This can be done running following SQL before and after update and then comparing the result sets (this show only given schema sequences):

SQL> spool test_schema_name_sequences_last_num_before.txt

SQL> set pages 120
SQL> set lines 120

SQL> select SEQUENCE_OWNER,SEQUENCE_NAME,LAST_NUMBER from dba_sequences where SEQUENCE_OWNER = '<TEST_SCHEMA_NAME'>) order by 2;

SQL> spool off;


# Again after changes:

SQL> spool test_schema_name_sequences_last_num_after.txt

SQL> set pages 120
SQL> set lines 120
SQL> select SEQUENCE_OWNER,SEQUENCE_NAME,LAST_NUMBER from dba_sequences where SEQUENCE_OWNER = '<TEST_SCHEMA_NAME'>) order by 2;
SQL> spool off;


NOTE! Remember that last_number column value is aware of cache. If you have cache 20 last_number is increased 20 every time next cache set is taken.  And because of this you can't depend only this column when cache is used you might need also check current_value from sequence. If cache is not used at all then last_number value is increased only when next value is taken from sequence and last_number is same as current_value.


If you want to get schema sequences creation clauses out of the database you can do it  with following SQL:
SQL> spool test_schema_name_sequences_ddl.txt

SQL> set pages 120
SQL> set lines 120 
SQL> select dbms_metadata.get_ddl('SEQUENCE',ds.sequence_name,ds.sequence_owner) from dba_sequences ds where ds.sequence_owner='<TEST_SCHEMA_NAME>';

SQL> spool off;

Oracle Statspack usage

Statspack is tool for performance monitoring and reporting.
New versions of Oracle (10 and 11) provides AWR (Automatic Workload Repository) reports with more detailed statistics than statspack.
But sometimes you might still need statspack for example with Oracle SE version databases.

Here is how you can get statspack report out of your database (First create tablespace for statspack before start install):

1. Install statspack (this creates PERFSTAT schema and this asks you to give tablespace name for statspack)(Run as oracle user (and SYS)):
cd $ORACLE_HOME/rdbms/admin
sqlplus "/ as sysdba" @spcreate.sql   


2. Take snapshots for statspack report (you need at least 2 snapshots to generate report):
sqlplus perfstat/<password>
exec statspack.snap;   

You can also give detail levels for snapshots (Default level is 5). Levels vary between Oracle versions.
Oracle 11.2 gives following levels:
SQL> select * from stats$level_description;

SNAP_LEVEL
----------
DESCRIPTION
------------------------------------------------------------------------------------------------------------------------
         0
This level captures general statistics, including rollback segment, row cache, SGA, system events, background events, session events, system statistics, wait statistics, lock statistics, and Latch information

         5
This level includes capturing high resource usage SQL Statements, along with all data captured by lower levels

         6
This level includes capturing SQL plan and SQL plan usage information for high resource usage SQL Statements, along with all data captured by lower levels

         7
This level captures segment level statistics, including logical and physical reads, row lock, itl and buffer busy waits, along with all data captured by lower levels

        10
This level includes capturing Child Latch statistics, along with all data captured by lower levels
----------------------------------

Take snapshot with certain level:
sqlplus perfstat/<password>
exec statspack.snap(i_snap_level=>10);

It is also possible to collect snapshots automatically via dbms_jobs (spauto.sql script) or with your own cron script.


3. Generate statspack report (This list available snapshots and asks you to give begin and end snapshot for report):
sqlplus perfstat/<password>
@?/rdbms/admin/spreport

You can also check available snapshots from here:
select SNAP_ID, SNAP_TIME from STATS$SNAPSHOT;

If you need a help for statspack report analyzing check these:
statspackanalyzer.com
 http://filebank.orapub.com/cgi-bin/quickUOWTBA.cgi


4. Purge statspack snapshots (You can purge old or all statspack snapshots from database):
Purge 10 days older snapshots:
sqlplus perfstat/<password>
exec statspack.purge(sysdate-10);



5. If you want to remove statspack schema and data from your database you can do it this way (Run as oracle user (and SYS)):
cd $ORACLE_HOME/rdbms/admin
sqlplus "/ as sysdba" @spdrop.sql      

Oracle Windows OPatch Prerequisite check "CheckActiveFilesAndExecutables" (Following files are active) failed.

When you are running Oracle OPatch patches in Windows environment you might get following error:
Running prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:

Following files are active :
C:\oracle\product\10.2.0\db_1\bin\oci.dll 
.
.
.

Patching guide says that before patching you should stop all Oracle services along with the "Distributed Transaction Coordinator" service. But sometimes this is not enough and you get "ActiveFiles" error when trying to run OPatch.

Usually problem is Windows Management Instrumentation (WMI) service. Restart this service from Services tool or you can also end process "WMIPrvSE.exe" from Task Manager (it will restart automatically). Then try to run OPatch again.

From following MOS (My Oracle Support) document you can find more help to this error:
Files in Use errors, when applying patches on Windows [ID 418479.1]

There is also Process Explorer tool which you can use to check which process is using certain ddl's:
http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

MySQL flush logs; gives: ERROR 1105 (HY000): Unknown error

If you trying to run "flush logs;" from mysql or with mysqladmin you'll get following error message:
ERROR 1105 (HY000): Unknown error

There are bugs that might affect like this but usually first thing to do with this is check log paths permissions.

1. First check what log directories your MySQL is using:
mysql> show global variables like '%log%';

You can also use more specific checks like:
This show error log:
mysql> show global variables like '%err%';
or for general log:
mysql> show global variables like '%general%';
or for slow log:
mysql> show global variables like '%slow%';


For example:

mysql> show global variables like '%err%';
+--------------------+----------------------+
| Variable_name      | Value                |
+--------------------+----------------------+
| error_count        | 0                    |
| log_error          | /data/mysql/logs/mysqld_error.log |
| max_connect_errors | 10                   |
| max_error_count    | 64                   |
| slave_skip_errors  | OFF                  |
+--------------------+----------------------+
5 rows in set (0.00 sec)



2. After you know log paths check that mysql user (or user that run mysqld if it is other than mysql.) have permissions to write in those directories (remember to check all log paths you are using).

ls -latr /data00/mysql/

drwxr-xr-x 14 mysql mysql 4096 Mar 26 06:46 logs

If there is not enough permissions then add them and the error is gone. If permissions are correct then you are probably hitting the bug.

Oracle Cluster Verification Utility (cluvfy) tracing.

If you get errors from Cluster Verification Utility (cluvfy) then you might need to get trace log out of it.
For example if you are creating Service Request about cluvfy into My Oracle Support you might add trace log into SR.

How to get trace log from cluvfy (run these as oracle user) :

1. Create directory for log:
mkdir /tmp/cvutrace

2. Set ENV variables:
export CV_TRACELOC=/tmp/cvutrace
export SRVM_TRACE=true
export SRVM_TRACE_LEVEL=1

3. Run wanted cluvfy check:
runcluvfy.sh .....

For example:
/data/install/grid/runcluvfy.sh stage -pre crsinst -n node1,node2 -r 11gR2 -verbose

After the cluvfy run is completed you'll have its trace log in the directory you created.
Copy it and add into the SR.

More info from MOS (My Oracle Support):
MOS ID 986822.1