Oracle 11gR2 RAC TNS-12542: TNS:address already in use 故障一例

原创: 2016-06-07 17:27:13828浏览

一．问题描述

一朋友在生产库的一个节点上做了一个非常危险的操作，然后该节点的数据库监听出现了异常。

数据库是2个节点的RAC，，版本11.2.0.3。

[grid@dave-db1 trace]$ lsnrctl status LISTENER

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 22-FEB-2013 18:40:27

Copyright (c) 1991, 2011, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date 22-FEB-2013 18:00:03
Uptime 0 days 0 hr. 40 min. 23 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /oracle/11.2.0.3/grid/network/admin/listener.ora
Listener Log File /oracle/app/grid/diag/tnslsnr/dave-db1/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
The listener supports no services
The command completed successfully

此处显示：The listener supports no services

[grid@dave-db1 trace]$ lsnrctl status LISTENER_SCAN1

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 22-FEB-2013 18:40:46

Copyright (c) 1991, 2011, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias LISTENER_SCAN1
Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date 22-FEB-2013 17:48:19
Uptime 0 days 0 hr. 52 min. 27 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /oracle/11.2.0.3/grid/network/admin/listener.ora
Listener Log File /oracle/11.2.0.3/grid/log/diag/tnslsnr/dave-db1/listener_scan1/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
The listener supports no services
The command completed successfully

手工注册后，问题依旧：

SQL>alter system register;

节点2上的监听是正常的：

[oracle@dave-db2 admin]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 22-FEB-2013 18:38:06

Copyright (c) 1991, 2011, Oracle. All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date 22-FEB-2013 17:30:45
Uptime 0 days 1 hr. 7 min. 21 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /oracle/11.2.0.3/grid/network/admin/listener.ora
Listener Log File /oracle/app/grid/diag/tnslsnr/dave-db2/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.106.14)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "DAVE" has 1 instance(s).
Instance "dave2", status READY, has 1 handler(s) for this service...
Service "oradb" has 1 instance(s).
Instance "dave2", status READY, has 1 handler(s) for this service...
Service "oradbXDB" has 1 instance(s).
Instance "dave2", status READY, has 1 handler(s) for this service...
The command completed successfully

--使用PS 查看监听的进程：

[grid@dave-db1 trace]$ ps -ef|grep LISTENER
grid 421 1 0 17:48 ? 00:00:00 /oracle/11.2.0.3/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid 1435 1 0 18:00 ? 00:00:00 /oracle/11.2.0.3/grid/bin/tnslsnr LISTENER -inherit
grid 4482 1333 0 18:39 pts/1 00:00:00 grep LISTENER
grid 14380 1 0 2012 ? 00:02:56 /oracle/11.2.0.3/grid/bin/tnslsnr LISTENER -inherit
grid 15075 1 0 2012 ? 00:03:07 /oracle/11.2.0.3/grid/bin/tnslsnr LISTENER_SCAN1 –inherit

注意1：

这里是有问题的，因为监听的进程出现了2次。我们继续看。

[grid@dave-db2 admin]$ crsctl stat resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE dave-db1
ONLINE ONLINE dave-db2
ora.LISTENER.lsnr
ONLINE INTERMEDIATE dave-db1 Not All Endpoints Registered
ONLINE ONLINE dave-db2
ora.asm
ONLINE ONLINE dave-db1 Started
ONLINE ONLINE dave-db2 Started
ora.gsd
OFFLINE OFFLINE dave-db1
OFFLINE OFFLINE dave-db2
ora.net1.network
ONLINE ONLINE dave-db1
ONLINE ONLINE dave-db2
ora.ons
ONLINE ONLINE dave-db1
ONLINE ONLINE dave-db2
ora.registry.acfs
ONLINE ONLINE dave-db1
ONLINE ONLINE dave-db2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE INTERMEDIATE dave-db1 Not All Endpoints Registered
ora.cvu
1 ONLINE ONLINE dave-db1
ora.dave-db1.vip
1 ONLINE ONLINE dave-db1
ora.dave-db2.vip
1 ONLINE ONLINE dave-db2
ora.oc4j
1 ONLINE ONLINE dave-db1
ora.oradb.db
1 ONLINE ONLINE dave-db2 Open
2 ONLINE ONLINE dave-db1 Open
ora.scan1.vip
1 ONLINE ONLINE dave-db1