raid强制上线后数据库无法启动故障处理
联系:手机/微信(+86 13429648788) QQ(107644445)
作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
由于raid掉盘过多,强制raid上线,然后启动数据库报以下错误
Mon Apr 19 23:19:28 2021 ALTER DATABASE OPEN Beginning crash recovery of 1 threads parallel recovery started with 15 processes
Started redo scan Completed redo scan read 106750 KB redo, 9080 data blocks need recovery
Mon Apr 19 23:19:45 2021 Slave exiting with ORA-1115 exception Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p000_3277 .trc:
ORA-01115: IO error reading block from file 9 (block # 339)
ORA-01110: data file 9: '/u01/app/oracle/oradata/orcl/dev02.dbf'
ORA-27072: File I /O error
Additional information: 4 Additional information: 326 Additional information: 24576 ORA-27072: File I /O error
Additional information: 4 Additional information: 326 Additional information: 24576 ORA-27072: File I /O error
Additional information: 4 Additional information: 326 Additional information: 24576 ORA-27072: File I /O error
Additional information: 4 Additional information: 326 Additional information: 24576 ORA-27072: File I /O error
Additional information: 4 Additional information: 326 Additional information: 24576 ORA-27072: File I /O error
Additional information: 4 Additional information: 326 Additional information: 24576 ORA-27072: File I /O error
Additional information: 4 Additional information: 326 Additional information: 24576 ORA-27072: File I /O error
Additional information: 4 Additional information: 326 Additional information: 24576 ORA-27072: File I Mon Apr 19 23:19:45 2021 Aborting crash recovery due to slave death, attempting serial crash recovery Beginning crash recovery of 1 threads Started redo scan Completed redo scan read 106750 KB redo, 9080 data blocks need recovery
Aborting crash recovery due to error 1115 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_3275 .trc:
ORA-01115: IO error reading block from file 9 (block # 329)
ORA-01110: data file 9: '/u01/app/oracle/oradata/orcl/dev02.dbf'
ORA-1115 signalled during: ALTER DATABASE OPEN... |
错误提示比较明显IO error,结合客户强行上线raid的操作,比较明显是由于底层io问题导致该错误,直接对此文件dbv检查
[oracle@database orcl]$ dbv file =dev02.dbf
DBVERIFY: Release 11.2.0.4.0 - Production on Mon Apr 19 23:59:03 2021 Copyright (c) 1982, 2011, Oracle and /or its affiliates. All rights reserved.
DBVERIFY - Verification starting : FILE = /u01/app/oracle/oradata/orcl/dev02 .dbf
DBV-00600: Fatal Error - [28] [27061] [0] [0] |
对于此类情况,通过工具进行处理
DUL> copy file from /u01/app/oracle/oradata/orcl/dev02 .dbf to /oradata/dev02 .dbf
starting copy datafile '/u01/app/oracle/oradata/orcl/dev02.dbf' to '/oradata/dev02.dbf'
read data error from file '/u01/app/oracle/oradata/orcl/dev02.dbf' .error message:Input /output error
read block # error: 303
read data error from file '/u01/app/oracle/oradata/orcl/dev02.dbf' .error message:Input /output error
read block # error: 304
read data error from file '/u01/app/oracle/oradata/orcl/dev02.dbf' .error message:Input /output error
read block # error: 329
datafile copy completed with 2 block error. |
dbv校验文件
[oracle@database oradata]$ dbv file =dev02.dbf
DBVERIFY: Release 11.2.0.4.0 - Production on Tue Apr 20 00:28:31 2021 Copyright (c) 1982, 2011, Oracle and /or its affiliates. All rights reserved.
DBVERIFY - Verification starting : FILE = /oradata/dev02 .dbf
Page 303 is marked corrupt Corrupt block relative dba: 0x0240012f ( file 9, block 303)
Completely zero block found during dbv: Page 304 is marked corrupt Corrupt block relative dba: 0x02400130 ( file 9, block 304)
Completely zero block found during dbv: Page 329 is marked corrupt Corrupt block relative dba: 0x02400149 ( file 9, block 329)
Completely zero block found during dbv: DBVERIFY - Verification complete Total Pages Examined : 3932160 Total Pages Processed (Data) : 3213723 Total Pages Failing (Data) : 0 Total Pages Processed (Index): 714294 Total Pages Failing (Index): 0 Total Pages Processed (Other): 4139 Total Pages Processed (Seg) : 0 Total Pages Failing (Seg) : 0 Total Pages Empty : 1 Total Pages Marked Corrupt : 3 Total Pages Influx : 0 Total Pages Encrypted : 0 Highest block SCN : 85078875 (6.85078875) |
通过对io error的文件进行处理,最终损坏三个block,最大限度抢救数据.使用被恢复出来的文件,尝试open库遭遇以下错误
SQL> alter database open resetlogs;
alter database open resetlogs
* ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [6], [85035771], [6], [85084136], [12583040], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [2662], [6], [85035770], [6], [85084136], [12583040], [], [], [], [], [], [] ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00600: internal error code, arguments: [2662], [6], [85035764], [6], [85084136], [12583040], [], [], [], [], [], [] Process ID: 6733 Session ID: 570 Serial number: 3 |
ora-600 2662这个错误比较明显,处理文件头scn,继续open库
SQL> alter database open ;
alter database open
* ERROR at line 1:
ORA-03113: end - of -file on communication channel
Process ID: 6840 Session ID: 570 Serial number: 3 |
查看alert日志信息
Tue Apr 20 01:22:27 2021 alter database open upgrade
Beginning crash recovery of 1 threads parallel recovery started with 15 processes
Started redo scan Completed redo scan read 1 KB redo, 3 data blocks need recovery
Started redo application at Thread 1: logseq 1, block 3
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0 Mem # 0: /u01/app/oracle/oradata/orcl/redo01.log
Completed redo application of 0.00MB Completed crash recovery at Thread 1: logseq 1, block 5, scn 25854859541
3 data blocks read , 3 data blocks written, 1 redo k-bytes read
Tue Apr 20 01:22:28 2021 Thread 1 advanced to log sequence 2 (thread open )
Thread 1 opened at log sequence 2 Current log # 2 seq# 2 mem# 0: /u01/app/oracle/oradata/orcl/redo02.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Apr 20 01:22:28 2021 SMON: enabling cache recovery [6840] Successfully onlined Undo Tablespace 2. Undo initialization finished serial:0 start:5902014 end:5905574 diff :3560 (35 seconds)
Dictionary check beginning Dictionary check complete Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery Database Characterset is ZHS16GBK Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824 .trc (incident=63970):
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], [] Incident details in : /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_63970/orcl_smon_6824_i63970 .trc
Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details.
ORACLE Instance orcl (pid = 14) - Error 600 encountered while recovering transaction (24, 2) on object 89023.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824 .trc:
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], [] Tue Apr 20 01:22:38 2021 ORACLE Instance orcl (pid = 14) - Error 600 encountered while recovering transaction (63, 3) on object 89023.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824 .trc:
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], [] Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824 .trc (incident=63974):
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], [] Incident details in : /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_63974/orcl_smon_6824_i63974 .trc
Tue Apr 20 01:22:55 2021 PMON (ospid: 6798): terminating the instance due to error 474 |
这个错误是比较常见的错误,参考:ORACLE Instance XFF (pid = 18) – Error 600 encountered while recovering transaction ,通过处理之后,数据库open成功
SQL> startup mount pfile= '/tmp/pfile' ;
ORACLE instance started. Total System Global Area 1603411968 bytes
Fixed Size 2253664 bytes
Variable Size 1023413408 bytes
Database Buffers 570425344 bytes
Redo Buffers 7319552 bytes Database mounted.
SQL> recover database ;
Media recovery complete. SQL> alter database open ;
Database altered.
|
后续安排逻辑导出,导入新库
- aix平台 ORA-01115 ORA-01110 ORA-27067 故障恢复
- ORA-600 kcratr_scan_lastbwr 恢复
- ORA-00600 dbkif_find_next_record_1
- 非归档数据库异常恢复一例
- ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], []
- ORA-00333 故障恢复
- ORA-600 999 异常恢复
- ORA-00600 kcratr_scan_rc
- 硬件故障数据库异常恢复
- ORA-00600[17182],ORA-00600[25027],ORA-00600[kghfrempty:ds]故障处理
- 分享一次ORA-01113 ORA-01110故障处理过程
- 在数据库恢复遭遇ORA-07445 kgegpa错误