Reference: http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.rsct.doc/rsct_aix5l53/bl5dia0216.html
Nov 4 12:51:51 nodename_4 RMCdaemon[23836]: (Recorded using libct_ffdc.a cv 2):::Error ID: 824….5pQw8/DqU1.X8Nc/……………….:::Reference ID: :::Template ID:
0:::Details File: :::Location: RSCT,rmcd.c,1.52,211 :::RMCD_INFO_0_ST The daemon is started.
Nov 4 12:51:51 nodename_4 RMCdaemon[23836]: (Recorded using libct_ffdc.a cv 2):::Error ID: 822….5pQw8/ZxU1.X8Nc/……………….:::Reference ID: :::Template ID:
0:::Details File: :::Location: RSCT,rmcd_env.c,1.36,123 :::RMCD_2610_101_ER Internal error. Error data 1 0000001c Error data 2 00000000 Error data 3 2641-0
37 The saved node identifiers do not agree.
Nov 4 12:51:51 nodename_4 srcmstr: src_error=-9035, errno=0, module=’srchevn.c’@line:’252′, 0513-035 The ctrmc Subsystem ended abnormally. SRC will try and restart it.
Nov 4 12:51:51 nodename_4 RMCdaemon[23839]: (Recorded using libct_ffdc.a cv 2):::Error ID: 824….5pQw8/wlY1.X8Nc/……………….:::Reference ID: :::Template ID:
0:::Details File: :::Location: RSCT,rmcd.c,1.52,211 :::RMCD_INFO_0_ST The daemon is started.
Nov 4 12:51:51 nodename_4 RMCdaemon[23839]: (Recorded using libct_ffdc.a cv 2):::Error ID: 822….5pQw8/SsY1.X8Nc/……………….:::Reference ID: :::Template ID:
0:::Details File: :::Location: RSCT,rmcd_env.c,1.36,123 :::RMCD_2610_101_ER Internal error. Error data 1 0000001c Error data 2 00000000 Error data 3 2641-0
37 The saved node identifiers do not agree.
Nov 4 12:51:51 nodename_4 srcmstr: src_error=-9035, errno=0, module=’srchevn.c’@line:’252′, 0513-035 The ctrmc Subsystem ended abnormally. SRC will try and restart it.
Nov 4 12:51:51 nodename_4 RMCdaemon[23841]: (Recorded using libct_ffdc.a cv 2):::Error ID: 824….5pQw8/pVc1.X8Nc/……………….:::Reference ID: :::Template ID:
0:::Details File: :::Location: RSCT,rmcd.c,1.52,211 :::RMCD_INFO_0_ST The daemon is started.
Nov 4 12:51:51 nodename_4 RMCdaemon[23841]: (Recorded using libct_ffdc.a cv 2):::Error ID: 822….5pQw8/ncc1.X8Nc/……………….:::Reference ID: :::Template ID:
0:::Details File: :::Location: RSCT,rmcd_env.c,1.36,123 :::RMCD_2610_101_ER Internal error. Error data 1 0000001c Error data 2 00000000 Error data 3 2641-0
37 The saved node identifiers do not agree.
Nov 4 12:51:51 nodename_4 srcmstr: src_error=-9020, errno=0, module=’srchevn.c’@line:’409′, 0513-020 The ctrmc Subsystem did not end normally. The subsystem respawn li
mit has been exceeded. Check the Subsystem and restart it manually.
fcslogrpt /var/log/messages
/usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc
On this node:
# cat /etc/ct_node_id
e30b82d7d977ae93
# cat /var/ct/cfg/ct_node_id
2fa704ea5403f37b
(Note: /var partition was completely corrupted, so we copied /var from another node in the cluster, to this one, thus the above 2 files are not matching each other)
# cp /var/ct/cfg/ct_node_id /var/ct/cfg/BACK_ct_node_id
# cp /etc/ct_node_id /var/ct/cfg/ct_node_id
# /etc/init.d/ctrmc status
Subsystem Group PID Status
ctrmc rsct inoperative
# /etc/init.d/ctrmc start
0513-059 The ctrmc Subsystem has been started. Subsystem PID is 23963.
# /etc/init.d/ctrmc status
Subsystem Group PID Status
ctrmc rsct 23963 active
/usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc
Management Domain Status: Management Control Points
O A 0xe2ae70929d8e2b07 0001 192.168.5.1
# lssrc -a
Subsystem Group PID Status
ctrmc rsct 23963 active
IBM.CSMAgentRM rsct_rm 24017 active
IBM.ERRM rsct_rm 24020 active
IBM.AuditRM rsct_rm 24059 active
ctcas rsct inoperative
IBM.HostRM rsct_rm inoperative
IBM.ConfigRM rsct_rm inoperative
IBM.SensorRM rsct_rm inoperative
IBM.FSRM rsct_rm inoperative
IBM.LPRM rsct_rm inoperative