1:周末遭遇停电,新配置的内网测试数据库active dataguard 环境自动关闭,今早启动主库后发现报错如下 [Oracle@db1 ~]$ sqlplus
1:周末遭遇停电,新配置的内网测试数据库active dataguard 环境自动关闭,今早启动主库后发现报错如下
[Oracle@db1 ~]$ sqlplus /nolog
SQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 17 17:00:57 2013
Copyright (c) 1982, 2009, Oracle. All rights reserved.
SQL> conn /as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 1.3362E 10 bytes
Fixed Size 2217952 bytes
Variable Size 6777997344 bytes
Database Buffers 6576668672 bytes
Redo Buffers 4960256 bytes
Database mounted.
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1],
[6], [301353], [301354], [], [], [], [], [], [], []
2:第一反应先recover下database,再尝试拉起数据库,问题依旧
SQL> recover database;
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1],
[6], [301353], [301354], [], [], [], [], [], [], []
3:于是看alert日志和相关的trace文件,综合判断可能控制文件出现问题
[oracle@db1 ~]$ tail -f alert_db.log
Errors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_ora_32462.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6], [301353], [301354], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_ora_32462.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6], [301353], [301354], [], [], [], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN...
Trace dumping is performing id=[cdmp_20130617170117]
Mon Jun 17 17:02:13 2013
Sweep [inc][24153]: completed
Sweep [inc2][24153]: completed
Mon Jun 17 17:02:38 2013
ALTER DATABASE RECOVER database
Media Recovery Start
started logmerger process
Parallel Media Recovery started with 4 slaves
Mon Jun 17 17:02:38 2013
Recovery of Online Redo Log: Thread 1 Group 3 Seq 6 Reading mem 0
Mem# 0: /u01/app/oracle/oradata/DB/onlinelog/o1_mf_3_8vpmjgoq_.log
Mem# 1: /u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_3_8vpmkm9x_.log
Media Recovery Complete (db)
Completed: ALTER DATABASE RECOVER database
Mon Jun 17 17:02:54 2013
alter database open
Beginning crash recovery of 1 threads
parallel recovery started with 3 processes
Started redo scan
Completed redo scan
read 152 KB redo, 0 data blocks need recovery
Errors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_ora_32462.trc (incident=24154):
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6], [301353], [301354], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/db1/db/incident/incdir_24154/db_ora_32462_i24154.trc
Mon Jun 17 17:02:55 2013
Trace dumping is performing id=[cdmp_20130617170255]
Aborting crash recovery due to error 600
Errors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_ora_32462.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6], [301353], [301354], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_ora_32462.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6], [301353], [301354], [], [], [], [], [], [], []
ORA-600 signalled during: alter database open...
Mon Jun 17 17:03:13 2013
Sweep [inc][24154]: completed
Sweep [inc2][24154]: completed
4:于是生成控制文件trace控制脚本对控制文件执行恢复
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
MOUNTED
SQL> alter database backup controlfile to trace as '/tmp/1.ctl';
Database altered.
[oracle@db1 ~]$ cat /tmp/1.ctl
CREATE CONTROLFILE REUSE DATABASE "DB" NORESETLOGS FORCE LOGGING ARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 8
MAXLOGHISTORY 292
LOGFILE
GROUP 1 (
'/u01/app/oracle/oradata/DB/onlinelog/o1_mf_1_8vpmdkl9_.log',
'/u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_1_8vpmdryt_.log'
) 大小 512M 块大小 512,
第 2 组 (
'/u01/app/oracle/oradata/DB/onlinelog/o1_mf_2_8vpmfqmw_.log',
'/u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_2_8vpmhk2z_.log'
) 大小 512M 块大小 512,
第 3 组 (
'/u01/app/oracle/oradata/DB/onlinelog/o1_mf_3_8vpmjgoq_.log',
'/u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_3_8vpmkm9x_.log'
) 大小 512M 块大小 512
--备用日志文件
-- 第 4 组 (
-- '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_4_8vpq6nvy_.log',
-- '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_4_8vpq7wk8_.log'
-- ) 大小 512M 块大小 512,
--第 5 组 (
-- '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_5_8vpqbh6s_.log',
-- '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_5_8vpqcmbj_.log'
-- ) 大小 512M 块大小 512,
-- 第 6 组 (
-- '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_6_8vpqf3rz_.log',
-- '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_6_8vpqfv5w_.log'
-- ) 大小 512M 块大小 512,
-- 第 7 组 (
-- '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_7_8vpqgw0j_.log',
-- '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_7_8vpqhcql_.log'
-- ) 大小 512M 块大小 512
数据文件
'/u01/app/oracle/oradata/DB/datafile/o1_mf_system_8vpm8hf3_.dbf',
'/u01/app/oracle/oradata/DB/datafile/o1_mf_sysaux_8vpm8hjq_.dbf',
'/u01/app/oracle/oradata/DB/datafile/o1_mf_undotbs1_8vpm8hl4_.dbf',
'/u01/app/oracle/oradata/DB/datafile/o1_mf_users_8vpm8hn1_.dbf'
字符集ZHS16GBK;
SQL>立即关闭
ORA-01109: 数据库未打开
数据库已卸载。
ORACLE 实例关闭。
SQL>启动 nomount
ORACLE 实例已启动。
总系统全局区域 1.3362E 10 字节
固定大小 2217952 字节
可变大小 6777997344 字节
数据库缓冲区 6576668672 字节
重做缓冲区 4960256 字节
SQL> @/tmp/1.ctl;
已创建控制文件。
5:重建控制文件后,对数据库进行恢复,,成功打开数据库
SQL>改变数据库打开;
更改数据库打开
*
第 1 行错误:
ORA-01113:文件 1 需要介质恢复
ORA-01110: 数据文件 1:
'/u01/app/oracle/oradata/DB/datafile/o1_mf_system_8vpm8hf3_.dbf'
SQL>恢复数据库;
媒体恢复完成。
SQL>改变数据库打开;
数据库已更改。
SQL>从 v$database 选择 open_mode,database_role;
OPEN_MODE DATABASE_ROLE
-------------------------------- ----------------
读写 初级
6:通过观察alert日志,添加临时表空间数据文件
[oracle@db1 ~]$ tail -f Alert_db.log
文件 /u01/app/oracle/diag/rdbms/db1/db/trace/db_m001_381.trc 中的错误:
ORA-25153:临时表空间为空
2013 年 6 月 17 日星期一 17:17:25
文件 /u01/app/oracle/diag/rdbms/db1/db/trace/db_j007_400.trc 中的错误:
ORA-25153:临时表空间为空
文件 /u01/app/oracle/diag/rdbms/db1/db/trace/db_j007_400.trc 中的错误:
ORA-12012:自动执行作业 12696 时出错
ORA-25153:临时表空间为空
ORA-06512:位于“DBSNMP.BSLN_INTERNAL”,第 2073 行
ORA-06512: 在第 1 行
2013 年 6 月 17 日星期一 17:18:20
更改表空间临时添加临时文件
已完成:alter tablespace temp add tempfile
SQL>更改表空间临时添加临时文件;
表空间已更改。
7:观察物理数据库,发现出现如下问题
[root@db2 ~]# su - oracle
[oracle@db2 ~]$ sqlplus /nolog
SQL*Plus:2013 年 6 月 17 日星期一 17:18:50 发布 11.2.0.1.0 产品
版权所有 (c) 1982、2009,Oracle。 保留所有权利。
SQL> conn /as sysdba
已连接到空闲实例。
SQL>启动
ORACLE 实例已启动。
总系统全局区域 1.3362E 10 字节
固定大小 2217952 字节
Variable Size 6777997344 bytes
Database Buffers 6576668672 bytes
Redo Buffers 4960256 bytes
Database mounted.
ORA-10458: standby database requires recovery
ORA-01196: file 1 is inconsistent due to a failed media recovery session
ORA-01110: data file 1:
'/u01/app/oracle/oradata/DB2/datafile/o1_mf_system_08oc7c38_.dbf'
SQL> alter database open read only;
alter database open read only
*
ERROR at line 1:
ORA-10458: standby database requires recovery
ORA-01196: file 1 is inconsistent due to a failed media recovery session
ORA-01110: data file 1:
'/u01/app/oracle/oradata/DB2/datafile/o1_mf_system_08oc7c38_.dbf'
8:于是关闭physical standby,在主库上启动system表空间的热备份,拷贝相应的文件至从库指定位置,重新同步正常
SQL> shutdown immediate
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
SQL> alter tablespace system begin backup;
Tablespace altered.
[oracle@db1 datafile]$ pwd
/u01/app/oracle/oradata/DB/datafile
[oracle@db1 datafile]$ scp o1_mf_system_8vpm8hf3_.dbf root@db2:/tmp
SQL> alter tablespace system end backup;
Tablespace altered.