问题: 机房检修停电,电源恢复后数据库启动失败。
诊断: 尝试启动实例:
$ sqlplus “/as sysdba”
startup nomount —正常
alter database mount; –报控制文件无法找到
诊断到这里,有点经验的工程师就能判断出来了,控制文件没有找到从而导致数据库无法正常启动。
可询问得知,数据库在停电前是用shutdown immediate 被正常关闭的,重启后丢失的概率非常低。
检查文件系统:
$ df -k
Filesystem 1024-blocks Free %Used Iused %Iused Mounted on
/dev/hd4 2097152 1922460 9% 10429 3% /
/dev/hd2 5242880 3311020 37% 37782 5% /usr
/dev/hd9var 2097152 2037480 3% 560 1% /var
/dev/hd3 5242880 4978432 6% 606 1% /tmp
/dev/hd1 2097152 2066076 2% 384 1% /home
/dev/hd11admin 524288 523864 1% 5 1% /admin
/proc - - - - - /proc
/dev/hd10opt 2097152 1998684 5% 3236 1% /opt
/dev/software 10485760 10483832 1% 4 1% /software
/dev/oracle 52428800 32706128 38% 28135 1% /oracle
/dev/rmandatalv 208404480 87904044 58% 19 1% /rmandata
观察到文件系统没有MOUNT起来。
检查卷组是否被激活(事前配置都是自动激活的):
$ lsvg -o
backupvg –归档、备份卷组已激活
datavg –数据卷组已激活
rootvg
P590_lpar1/>lsvg -l datavg
datavg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
oradata jfs2 256 256 2 closed/syncd /oradata
loglv00 jfs2log 1 1 1 closed/syncd N/A
P590_lpar1/>lsvg -l backupvg
backupvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
archlv jfs2 800 800 2 closed/syncd /archlog
loglv01 jfs2log 1 1 1 open/syncd N/A
rmandatalv jfs2 1590 1590 2 open/syncd /rmandata
检查发现,卷组正常,文件系统不正常。
尝试mount文件系统:
P590_lpar1/>mount /oradata
Replaying log for /dev/oradata.
mount: 0506-324 Cannot mount /dev/oradata on /oradata: The media is not formatte
d or the format is not correct.
0506-342 The superblock on /dev/oradata is dirty. Run a full fsck to fix.
P590_lpar1/>mount /archlog
Replaying log for /dev/archlv.
mount: 0506-324 Cannot mount /dev/archlv on /archlog: The media is not formatted
or the format is not correct.
0506-342 The superblock on /dev/archlv is dirty. Run a full fsck to fix.
问题出在文件系统不能正常mont上:
文件系统一致性检查:
#fsck /oradata
The current volume is: /dev/oradata
Primary superblock is valid.
J2_LOGREDO:log redo processing for /dev/oradata
Primary superblock is valid.
*** Phase 1 – Initial inode scan
*** Phase 2 – Process remaining directories
*** Phase 3 – Process remaining files
*** Phase 4 – Check and repair inode allocation map
*** Phase 5 – Check and repair block allocation map
File system is clean.
Superblock is marked dirty; FIX? yes
All observed inconsistencies have been repaired.
#mount /oradata
正常mount成功。
同样地,/archlog也做同样的处理即可。
启动数据库正常。
总结:
系统配置和数据库自动启动配置等都正常,这次出问题的原因是存储没有正常的断电。
导致文件系统mount时不一致。
本次故障虽一切顺利,但是如果fsck无法正常,那后果将很严重,所有正常的运维工作也不可缺少。

