[RAC] Node 1 cannot start due to lack of cssfatal-CSS Tutorial-php.cn

Environment: AIX5.3+10.2.0.5 RAC

Scenario description: When RAC is restarted after shutting down, the first node cannot start, but the second node starts normally

Troubleshooting process:

1. Try to start the crs service of node 1

root# ./init.crs start crs

2. During the monitoring startup process, crs log

OCSSD.log log:

[CSSD]2014-01-16 09:27:54.730 >USER: Starting CSS daemon on node nxjcdb1, number1, in cluster crs_dljc

[clsdmt]Listening to(ADDRESS= (PROTOCOL=ipc)(KEY=nxjcdb1DBG_CSSD))

[CSSD]2014-01-16 09:27:54.790 [1]>TRACE: clssscmain: RT queuesetting: ON

[CSSD]2014-01-16 09 :27:55.081 [1]>TRACE: clssscmain: local-only setto false

[CSSD]2014-01-16 09:27:55.349 [1]>TRACE: clssnmReadNodeInfo: addednode 1 (nxjcdb1) to cluster

[ CSSD]2014-01-16 09:27:55.672 [1]>TRACE: clssnmReadNodeInfo: addednode 2 (nxjcdb2) to cluster

[ CSSD]2014-01-16 09:27:55.673 [1]> TRACE: clssnmInitNMInfo:Initialized with unique 1389835674

[CSSD]2014-01-16 09:27:55.704 [1]>TRACE: clssNMInitialize:Initializing with OCR id (1516675067)

[CSSD]2014-01-1 6 09 :27:55.705 [1029] >TRACE: clssnm_skgxninit: HACMP clusterware detected

[ CSSD]2014-01-16 09:27:56.822 [1]>TRACE: clssnmNMInitialize:misscount set to (30)

[ CSSD ]2014-01-16 09:27:56.900 [1]>TRACE: clssnmStartNM: reboottimeset to (3) sec

[ CSSD]2014-01-16 09:27:56.900 [1]>TRACE: clssnmNMInitialize: Networkheartbeat thresholds are: impending reconfig 15000 ms, reconfig start(misscount) 30000 ms

[CSSD]2014-01-16 09:27:57.108 [1]>TRACE: clssnmDiskStateChange: statefrom 1 to 2 disk (0//dev /rlvjc_voting)

[CSSD]2014-01-16 09:27:57.108 [1030]>TRACE: clssnmvDPT: spawned for disk0 (/dev/rlvjc_voting)

[CSSD]2014-01-16 09:27: 1030 //dev/rlvjc_voting)

[CSSD]2014-01-16 09:27:59.164 [1]>ERROR: Internal Error Information:

Category: 1234

Operation: scls_scr_setval

Location: open

Other : cant open file

Dep: 2

[CSSD]2014-01-16 09:27:59.164 [1]>ERROR: clssscSclsFatal: failure 8reading fatal mode

[CSSD]2014-01-16 09:27 :59.164 [1]>ERROR: ################################

[CSSD]2014 -01-16 09:27:59.164 [1]>ERROR: clssscExit: CSSD aborting from thread Main

[ CSSD]2014-01-16 09:27:59.164 [1]>ERROR: ###### ############################

→According to the error message, it is initially determined that OCSSD cannot start because node 1 cannot vote disk.

[ CSSD]--- DUMP GROCK STATE DB ---

[ CSSD]--- END OF GROCK STATE DUMP ---

[ CSSD]2014-01-16 09:27:59.169 [1030]> ;TRACE: clssnmvReadDskHeartbeat:read ALL for Joining

[CSSD]2014-01-16 09:27:59.169 [1030]>TRACE: clssnmvReadDskHeartbeat:node(2) is down. rcfg(2) wrtcnt(126947) LATS( 1038806686) Disk lastSeqNo(126947)

[CSSD]------- Begin Dump -------

[CSSD]

[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00..... ........

[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ... .............

[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863e0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............

[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100863f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............

[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086400 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............

[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086410 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............

[CSSD]2014-01-16 09:28:00.166 [1 ]>TRACE: 0x110086420 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 .....

[CSSD]2014-01-16 09: 28:00.166 [1]>TRACE: 0x110086430 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 .....

[CSSD]2014- 01-16 09:28:00.166 [1]>TRACE: 0x110086440 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............

[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086450 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ............. ...

[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086460 00 00 00 01 1008 61 98 - 00 00 00 01 10 c6 0b c0 ......a. ........

[CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086470 00 00 00 00 00 0000 01 - 00 00 00 00 00 02 00 03 ... .............

[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086480 00 00 00 01 1096 4a b0 - 00 00 00 00 00 00 00 00 ......J.........

[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086490 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 01 ................

[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864c0 00 00 00 00 0000 00 05 - 00 00 00 00 00 00 00 fa ................

[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864e0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x1100864f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086500 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.166 [1]>TRACE: 0x110086510 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086520 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086530 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086540 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086550 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086560 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086570 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086580 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086590 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865a0 00 00 00 00 0000 00 00 - 00 00 00 0e 00 00 00 24 ...............$

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865e0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100865f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086600 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086610 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086620 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086630 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086640 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086650 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086660 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086670 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086680 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x110086690 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866b0 00 00 10 00 0000 00 97 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866c0 00 00 00 01 105d f6 10 - 00 00 00 01 10 95 ca 50 .....].........P

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866d0 00 00 00 01 1096 0a 70 - 00 00 00 3c 00 00 00 00 .......p...<....

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866e0 00 00 00 01 1096 2a 90 - 00 00 00 01 00 00 00 01 ......*.........

[ CSSD]2014-01-16 09:28:00.167 [1]>TRACE: 0x1100866f0 00 00 00 28 0000 00 00 - 00 00 00 01 10 00 16 08 ...(............

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086700 00 00 00 00 0000 00 00 - 00 00 00 01 10 4c 1a 90 .............L..

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086710 00 00 00 01 104c 1a 50 - 00 00 00 01 10 08 67 18 .....L.P......g.

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086720 00 00 00 01 1008 67 18 - 00 00 00 00 00 00 00 00 ......g.........

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086730 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086740 00 00 00 01 10cf 76 b0 - 00 00 00 00 00 00 00 00 ......v.........

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086750 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086760 00 00 00 01 10cf 76 b0 - 00 00 00 00 00 00 00 00 ......v.........

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086770 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086780 00 00 01 00 0000 00 00 - 00 00 00 01 10 bd f7 90 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086790 00 00 00 01 104c 17 50 - 00 00 00 00 00 00 00 00 .....L.P........

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867a0 00 00 00 00 0000 00 00 - 00 00 00 03 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867b0 00 00 00 01 1000 09 78 - 00 00 00 01 00 00 00 00 .......x........

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867c0 00 00 00 00 0000 00 01 - 00 00 00 01 10 3e df b0 .............>..

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867d0 00 00 00 01 1046 49 90 - 00 00 00 01 10 46 4a 30 .....FI......FJ0

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867e0 00 00 00 01 1046 4c 90 - 6e 78 6a 63 64 62 31 00 .....FL.nxjcdb1.

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100867f0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086800 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086810 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086820 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086830 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086840 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086850 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086860 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086870 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086880 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x110086890 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.168 [1]>TRACE: 0x1100868e0 00 00 00 00 0000 00 00 - 6e 78 6a 63 64 62 31 2d ........nxjcdb1-

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100868f0 70 72 69 00 0000 00 00 - 00 00 00 00 00 00 00 00 pri.............

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086900 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086910 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086920 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086930 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086940 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086950 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086960 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086970 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086980 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086990 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869a0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869b0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869c0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869d0 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869e0 00 00 00 00 0000 00 00 - 2f 6f 72 61 63 6c 65 2f ......../oracle/

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x1100869f0 70 72 6f 64 7563 74 2f - 31 30 2e 32 2e 30 2f 63 product/10.2.0/c

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a00 72 73 5f 31 0000 00 00 - 00 00 00 00 00 00 00 00 rs_1............

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a10 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a20 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a30 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a40 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a50 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a60 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a70 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086a80 00 00 00 00 00 00 00 00 - 00 0000 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1]>TRACE: 0x110086a90 00 00 00 00 0000 00 00 - 00 00 00 00 00 00 00 00 ................

[ CSSD]2014-01-16 09:28:00.169 [1] >TRACE: 0x110086aa0 00 00 00 00 00 00 00 00 -00 00 00 00 00 00 00 00 ................

/dev/rlvjc_voting

3. 日志报错，节点1的crs服务无法启动。

根据报错信息，初步判定是因为节点1无法voting disk造成 OCSSD无法启动。

4. 问题定位

发现/etc/oracle/scls_scr/ballontt(主机名)/oracle路径下缺少cssfatal文件，此文件里只有“enable”一个单词

~cat cssfatal

Enable

5. 问题解决

手动添加该文件

~vi cssfatal

Enable

6. 成功启动节点1的CRS