ORA-00607 Error 607 encountered recovering transaction db_block_checking

It was fine morning but by afternoon, it was an ooopss !! usual DBA days :) .

In our production  Oracle EBS R12.1.3  PROD system, we received data corruption issues that caused the bounce SMON to go down and after which SMON could not recover the corrupt data block and instance could not be brought up.  DB alert log we found below error

ORACLE Instance EBSXX (pid = 14) - Error 607 encountered while recovering transaction (7, 5) on object 46989.
Errors in file /move/prod/db/tech_st/11.2.0/admin/EBSXX_ebsp-pkg/diag/rdbms/ebsp/EBSXX/trace/EBSXX_smon_16908566.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [6856], [0], [65], [], [], [], [], [], [], [], [], []
PMON (ospid: 21430580): terminating the instance due to error 474
Thu Oct 12 07:53:55 2017
opiodr aborting process unknown ospid (23396816) as a result of ORA-1092
Thu Oct 12 07:53:55 2017
ORA-1092 : opitsk aborting process
Thu Oct 12 07:53:55 2017
System state dump requested by (instance=1, osid=21430580 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /move/XXX/trace/EBSXX_diag_30999036_20171012075355.trc
Instance terminated by PMON, pid = 21430580

The root cause was identified as DB parameter db_block_checking value was not set. You can follow below steps to recover that database. Connect as SYSDBA and using existing spfile start the database in nomount option.

SQL> Startup nomount ;
SQL> Create pfile='/tmp/corrupt.ora' from spfile ;
SQL> Shutdown immediate;

Modify the corrupt.ora and add below line at the end of the file to recover SMON with trace details
event = '10513 trace name context forever, level 2'

SQL> Startup pfile='/tmp/corrupt.ora'

Oracle support also suggested another option to set below parameter dynamically to start DB

set "_smu_debug_mode"=1024 to stop Transaction recovery (it can be done dynamically):
alter system set "_smu_debug_mode"=1024;

The above step will allow the database to startup by making SMON to skip the recovery of corrupt data block. Below SQL will help to find more detail of block affected by data corruption issue.

SQL> select useg.segment_name, useg.segment_id, useg.tablespace_name, useg.status from dba_rollback_segs useg where useg.segment_id = '7';

SQL> select XIDUSN, XIDSLOT, XIDSQN, USED_UBLK, USED_UREC from gv$transaction where XIDUSN = 7;

Edit your init@SID.ora and add below

_CORRUPTED_ROLLBACK_SEGMENTS= _SYSSMU7_1302289624$

In our case , we were lucky that we just truncated the table and ran EBS concurrent program that successfully repopulated the table. But in most cases, you have to recover the table

The issue can be prevented by setting below parameter in database
Alter system set db_block_checking = MEDIUM scope=both;

NOTE : Setting the value for db_block_checking to TRUE or FULL might add overhead of 1% to 10% in overall performance issue.