CRS-4000: Command Start failed, or completed with errors. during start CRS in RAC cluster

 Unable to start CRS :
CRS-4000: Command Start failed, or completed with errors.



[root@rac1 bin]# ./crsctl stop cluster
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.DATA1.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.orcl.db' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATA2.dg' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'
CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2677: Stop of 'ora.DATA2.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.DATA1.dg' on 'rac1' succeeded
CRS-2679: Attempting to clean 'ora.DATA1.dg' on 'rac1'
CRS-2681: Clean of 'ora.DATA1.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1'
CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
[root@rac1 bin]#

[root@rac1 bin]#  ps -ef|grep d.bin
root     24016     1  0 01:43 ?        00:00:05 /u01/app/11.2.0/grid_1/bin/ohasd.bin reboot
grid     24180     1  0 01:43 ?        00:00:00 /u01/app/11.2.0/grid_1/bin/mdnsd.bin
grid     24191     1  0 01:43 ?        00:00:00 /u01/app/11.2.0/grid_1/bin/gpnpd.bin
grid     24202     1  0 01:44 ?        00:00:03 /u01/app/11.2.0/grid_1/bin/gipcd.bin
root     24218     1  1 01:44 ?        00:00:07 /u01/app/11.2.0/grid_1/bin/osysmond.bin
root     25860 22673  0 01:54 pts/1    00:00:00 grep d.bin

After try to start crs then got  below error 

[root@rac1 bin]# ./crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
[root@rac1 bin]#

[root@rac1 bin]# ./crsctl check  crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[root@rac1 bin]#

Try to start with below command

[root@rac1 bin]# ./crsctl start cluster
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
[root@rac1 bin]#


[root@rac1 bin]# ps -ef|grep d.bin
root     24016     1  0 01:43 ?        00:00:08 /u01/app/11.2.0/grid_1/bin/ohasd.bin reboot
grid     24180     1  0 01:43 ?        00:00:00 /u01/app/11.2.0/grid_1/bin/mdnsd.bin
grid     24191     1  0 01:43 ?        00:00:01 /u01/app/11.2.0/grid_1/bin/gpnpd.bin
grid     24202     1  0 01:44 ?        00:00:06 /u01/app/11.2.0/grid_1/bin/gipcd.bin
root     24218     1  1 01:44 ?        00:00:15 /u01/app/11.2.0/grid_1/bin/osysmond.bin
grid     25921     1  0 01:55 ?        00:00:03 /u01/app/11.2.0/grid_1/bin/ocssd.bin
root     25975     1  0 01:55 ?        00:00:01 /u01/app/11.2.0/grid_1/bin/octssd.bin
grid     25991     1  0 01:55 ?        00:00:01 /u01/app/11.2.0/grid_1/bin/evmd.bin
root     26113     1  1 01:55 ?        00:00:04 /u01/app/11.2.0/grid_1/bin/crsd.bin reboot
root     27141 22673  0 02:01 pts/1    00:00:00 grep d.bin
[root@rac1 bin]#