Oracle RAC Database 11.1.0.6监听故障案例
接到电话，客户的一套核心Oracle RAC数据库连接不上，连接时报无监听程序，客户的Oracle RAC版本为11.1.0.6，平台为AIX 6.1.05，
接到电话，客户的一套核心Oracle RAC数据库连接不上，连接时报无监听程序，客户的Oracle RAC版本为11.1.0.6，，平台为AIX 6.1.05，使用了IBM HACMP 5.5.0.8。
当我远程过去的时候，发现节点2已经没有任何oracle用户的进程，且concurrent的vg没有激活，HACMP的服务也offline。
另一个节点Oracle的实例是正常的，且有部分服务器进程依然在工作，但是本地监听器出现了故障，导致新的连接无法连接到实例，通过crs_stat -t看到两个实例的监听也都是OFFLINE状态。
在节点上并没有发现有LISTENER进程，且手动杀掉了所有的服务器进程，在oracle用户下启动监听时收到以下的报错：
$ lsnrctl start listener_cdfy740a
LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:09:09
Copyright (c) 1991, 2007, Oracle. All rights reserved.
Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...
TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production
System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
TNS-12542: TNS:address already in use
TNS-12560: TNS:protocol adapter error
TNS-00512: Address already in use
IBM/AIX RISC System/6000 Error: 67: Address already in use
Listener failed to start. See the error message(s) above...
10.107.64.1是该节点的vip地址，下面是RAC环境的hosts配置：
10.107.64.1 vip1
10.107.64.2 vip2
10.107.64.3 cdfy740a
10.107.64.4 cdfy740b
172.201.201.1 prv1
172.201.201.2 prv2
手动停掉该节点的nodeapps服务：
cdfy740a@root[/oracle/app/11.1.0/crs/bin]./srvctl stop nodeapps -n cdfy740a
成功停止后，VIP在主机层面已经消失：
cdfy740a@root[/oracle/app/11.1.0/crs/bin]ifconfig -a | more
en0: flags=1e080863,c0
inet 172.200.200.1 netmask 0xffffff00 broadcast 172.200.200.255
tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en1: flags=1e080863,c0
inet 172.201.201.1 netmask 0xffffff00 broadcast 172.201.201.255
tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en4: flags=5e080863,c0
inet 10.107.64.3 netmask 0xffffff00 broadcast 10.107.64.255
tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
lo0: flags=e08084b,c0
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
inet6 ::1%1/0
tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
再次启动节点nodeapps服务：
cdfy740a@root[/oracle/app/11.1.0/crs/bin]./srvctl start nodeapps -n cdfy740a
CRS-1006: No more members to consider
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:13:07
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Copyright (c) 1991, 2007, Oracle. All rights reserved.
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12542: TNS:address already in use
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-00512: Address already in use
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: IBM/AIX RISC System/6000 Error: 67: Address already in use
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Listener failed to start. See the error message(s) above...
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:13:08
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Copyright (c) 1991, 2007, Oracle. All rights reserved.
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vip1)(PORT=1521)))
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-00511: No listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: IBM/AIX RISC System/6000 Error: 79: Connection refused
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-00511: No listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: IBM/AIX RISC System/6000 Error: 79: Connection refused
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.3)(PORT=1521)(IP=FIRST)))
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-00511: No listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: IBM/AIX RISC System/6000 Error: 79: Connection refused
CRS-0215: Could not start resource 'ora.cdfy740a.LISTENER_CDFY740A.lsnr'.
之前使用lsnrctl status listener_cdfy740a查看监听器状态时也收到Connection refused的错误。
Hot AI Tools
Undress AI Tool
Undress images for free
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Handling character sets and collations issues in MySQL
Jul 08, 2025 am 02:51 AM
Character set and sorting rules issues are common when cross-platform migration or multi-person development, resulting in garbled code or inconsistent query. There are three core solutions: First, check and unify the character set of database, table, and fields to utf8mb4, view through SHOWCREATEDATABASE/TABLE, and modify it with ALTER statement; second, specify the utf8mb4 character set when the client connects, and set it in connection parameters or execute SETNAMES; third, select the sorting rules reasonably, and recommend using utf8mb4_unicode_ci to ensure the accuracy of comparison and sorting, and specify or modify it through ALTER when building the library and table.
Connecting to MySQL Database Using the Command Line Client
Jul 07, 2025 am 01:50 AM
The most direct way to connect to MySQL database is to use the command line client. First enter the mysql-u username -p and enter the password correctly to enter the interactive interface; if you connect to the remote database, you need to add the -h parameter to specify the host address. Secondly, you can directly switch to a specific database or execute SQL files when logging in, such as mysql-u username-p database name or mysql-u username-p database name
Implementing Transactions and Understanding ACID Properties in MySQL
Jul 08, 2025 am 02:50 AM
MySQL supports transaction processing, and uses the InnoDB storage engine to ensure data consistency and integrity. 1. Transactions are a set of SQL operations, either all succeed or all fail to roll back; 2. ACID attributes include atomicity, consistency, isolation and persistence; 3. The statements that manually control transactions are STARTTRANSACTION, COMMIT and ROLLBACK; 4. The four isolation levels include read not committed, read submitted, repeatable read and serialization; 5. Use transactions correctly to avoid long-term operation, turn off automatic commits, and reasonably handle locks and exceptions. Through these mechanisms, MySQL can achieve high reliability and concurrent control.
Managing Character Sets and Collations in MySQL
Jul 07, 2025 am 01:41 AM
The setting of character sets and collation rules in MySQL is crucial, affecting data storage, query efficiency and consistency. First, the character set determines the storable character range, such as utf8mb4 supports Chinese and emojis; the sorting rules control the character comparison method, such as utf8mb4_unicode_ci is case-sensitive, and utf8mb4_bin is binary comparison. Secondly, the character set can be set at multiple levels of server, database, table, and column. It is recommended to use utf8mb4 and utf8mb4_unicode_ci in a unified manner to avoid conflicts. Furthermore, the garbled code problem is often caused by inconsistent character sets of connections, storage or program terminals, and needs to be checked layer by layer and set uniformly. In addition, character sets should be specified when exporting and importing to prevent conversion errors
Using Common Table Expressions (CTEs) in MySQL 8
Jul 12, 2025 am 02:23 AM
CTEs are a feature introduced by MySQL8.0 to improve the readability and maintenance of complex queries. 1. CTE is a temporary result set, which is only valid in the current query, has a clear structure, and supports duplicate references; 2. Compared with subqueries, CTE is more readable, reusable and supports recursion; 3. Recursive CTE can process hierarchical data, such as organizational structure, which needs to include initial query and recursion parts; 4. Use suggestions include avoiding abuse, naming specifications, paying attention to performance and debugging methods.
Strategies for MySQL Query Performance Optimization
Jul 13, 2025 am 01:45 AM
MySQL query performance optimization needs to start from the core points, including rational use of indexes, optimization of SQL statements, table structure design and partitioning strategies, and utilization of cache and monitoring tools. 1. Use indexes reasonably: Create indexes on commonly used query fields, avoid full table scanning, pay attention to the combined index order, do not add indexes in low selective fields, and avoid redundant indexes. 2. Optimize SQL queries: Avoid SELECT*, do not use functions in WHERE, reduce subquery nesting, and optimize paging query methods. 3. Table structure design and partitioning: select paradigm or anti-paradigm according to read and write scenarios, select appropriate field types, clean data regularly, and consider horizontal tables to divide tables or partition by time. 4. Utilize cache and monitoring: Use Redis cache to reduce database pressure and enable slow query
Designing a Robust MySQL Database Backup Strategy
Jul 08, 2025 am 02:45 AM
To design a reliable MySQL backup solution, 1. First, clarify RTO and RPO indicators, and determine the backup frequency and method based on the acceptable downtime and data loss range of the business; 2. Adopt a hybrid backup strategy, combining logical backup (such as mysqldump), physical backup (such as PerconaXtraBackup) and binary log (binlog), to achieve rapid recovery and minimum data loss; 3. Test the recovery process regularly to ensure the effectiveness of the backup and be familiar with the recovery operations; 4. Pay attention to storage security, including off-site storage, encryption protection, version retention policy and backup task monitoring.
Optimizing complex JOIN operations in MySQL
Jul 09, 2025 am 01:26 AM
TooptimizecomplexJOINoperationsinMySQL,followfourkeysteps:1)EnsureproperindexingonbothsidesofJOINcolumns,especiallyusingcompositeindexesformulti-columnjoinsandavoidinglargeVARCHARindexes;2)ReducedataearlybyfilteringwithWHEREclausesandlimitingselected
