CRS-4000: Command Start failed, or completed with errors. during start CRS in RAC cluster

 Unable to start CRS :
CRS-4000: Command Start failed, or completed with errors.



[root@rac1 bin]# ./crsctl stop cluster
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.DATA1.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.orcl.db' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATA2.dg' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'
CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2677: Stop of 'ora.DATA2.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.DATA1.dg' on 'rac1' succeeded
CRS-2679: Attempting to clean 'ora.DATA1.dg' on 'rac1'
CRS-2681: Clean of 'ora.DATA1.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1'
CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
[root@rac1 bin]#

[root@rac1 bin]#  ps -ef|grep d.bin
root     24016     1  0 01:43 ?        00:00:05 /u01/app/11.2.0/grid_1/bin/ohasd.bin reboot
grid     24180     1  0 01:43 ?        00:00:00 /u01/app/11.2.0/grid_1/bin/mdnsd.bin
grid     24191     1  0 01:43 ?        00:00:00 /u01/app/11.2.0/grid_1/bin/gpnpd.bin
grid     24202     1  0 01:44 ?        00:00:03 /u01/app/11.2.0/grid_1/bin/gipcd.bin
root     24218     1  1 01:44 ?        00:00:07 /u01/app/11.2.0/grid_1/bin/osysmond.bin
root     25860 22673  0 01:54 pts/1    00:00:00 grep d.bin

After try to start crs then got  below error 

[root@rac1 bin]# ./crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
[root@rac1 bin]#

[root@rac1 bin]# ./crsctl check  crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[root@rac1 bin]#

Try to start with below command

[root@rac1 bin]# ./crsctl start cluster
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
[root@rac1 bin]#


[root@rac1 bin]# ps -ef|grep d.bin
root     24016     1  0 01:43 ?        00:00:08 /u01/app/11.2.0/grid_1/bin/ohasd.bin reboot
grid     24180     1  0 01:43 ?        00:00:00 /u01/app/11.2.0/grid_1/bin/mdnsd.bin
grid     24191     1  0 01:43 ?        00:00:01 /u01/app/11.2.0/grid_1/bin/gpnpd.bin
grid     24202     1  0 01:44 ?        00:00:06 /u01/app/11.2.0/grid_1/bin/gipcd.bin
root     24218     1  1 01:44 ?        00:00:15 /u01/app/11.2.0/grid_1/bin/osysmond.bin
grid     25921     1  0 01:55 ?        00:00:03 /u01/app/11.2.0/grid_1/bin/ocssd.bin
root     25975     1  0 01:55 ?        00:00:01 /u01/app/11.2.0/grid_1/bin/octssd.bin
grid     25991     1  0 01:55 ?        00:00:01 /u01/app/11.2.0/grid_1/bin/evmd.bin
root     26113     1  1 01:55 ?        00:00:04 /u01/app/11.2.0/grid_1/bin/crsd.bin reboot
root     27141 22673  0 02:01 pts/1    00:00:00 grep d.bin
[root@rac1 bin]#

How to change/Modify SCAN Listener port in oracle RAC

 CHANGE  SCAN LISTENER PORT IN RAC DATABASE
================================================

Step 1:- verify current status and port of scan listener 

[grid@rac1 ~]$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521
[grid@rac1 ~]$
[grid@rac1 ~]$
[grid@rac1 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node rac2
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node rac1
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node rac1
[grid@rac1 ~]$



Step 2:-  Change port number of the SCAN LISTENER

[grid@rac1 ~]$ srvctl modify scan_listener -p 1585
[grid@rac1 ~]$
[grid@rac1 ~]$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1585
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1585
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1585
[grid@rac1 ~]$

Step 3:- Restart SCAN LISTENER using below command 

[grid@rac1 ~]$ srvctl stop scan_listener
[grid@rac1 ~]$
[grid@rac1 ~]$ srvctl start scan_listener
[grid@rac1 ~]$
[grid@rac1 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node rac2
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node rac1
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node rac1
[grid@rac1 ~]$
[grid@rac1 ~]$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1585
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1585
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1585
[grid@rac1 ~]$
[grid@rac1 ~]$
[grid@rac1 ~]$
[grid@rac1 ~]$
[grid@rac1 ~]$
[grid@rac1 ~]$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1585
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1585
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1585
[grid@rac1 ~]$
[grid@rac1 ~]$

Step 4:- Modify remote_listener parameter in each rac instance  also change port number in  tns entries

Before database stop change 

alter system set remote_listener='rac-scan:1585' scope=both sid='*';

Bounce database
[grid@rac1 ~]$ srvctl stop database -d orcl
[grid@rac1 ~]$
[grid@rac1 ~]$ srvctl start database -d orcl
[grid@rac1 ~]$
[grid@rac1 ~]$
[grid@rac1 ~]$ srvctl status database -d orcl
Instance orcl1 is running on node rac1
Instance orcl2 is running on node rac2
[grid@rac1 ~]$
[grid@rac1 ~]$ lsnrctl status LISTENER_SCAN2

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 28-JAN-2017 23:41:14

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN2
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                28-JAN-2017 23:32:15
Uptime                    0 days 0 hr. 8 min. 59 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0/grid_1/network/admin/listener.ora
Listener Log File         /u01/app/11.2.0/grid_1/log/diag/tnslsnr/rac1/listener_scan2/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN2)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.110)(PORT=1585)))
Services Summary...
Service "orcl.localdomain.com" has 2 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
  Instance "orcl2", status READY, has 1 handler(s) for this service...
Service "orclXDB.localdomain.com" has 2 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
  Instance "orcl2", status READY, has 1 handler(s) for this service...
The command completed successfully
[grid@rac1 ~]$
[grid@rac1 ~]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 28-JAN-2017 23:41:23

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                28-JAN-2017 17:45:42
Uptime                    0 days 5 hr. 55 min. 41 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0/grid_1/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/rac1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.201)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.40)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "orcl.localdomain.com" has 1 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
Service "orclXDB.localdomain.com" has 1 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
The command completed successfully
[grid@rac1 ~]$

[grid@rac2 bin]$ lsnrctl status LISTENER_SCAN1

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 28-JAN-2017 23:45:01

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN1
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                28-JAN-2017 23:32:13
Uptime                    0 days 0 hr. 12 min. 48 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0/grid_1/network/admin/listener.ora
Listener Log File         /u01/app/11.2.0/grid_1/log/diag/tnslsnr/rac2/listener_scan1/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.112)(PORT=1585)))
Services Summary...
Service "orcl.localdomain.com" has 2 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
  Instance "orcl2", status READY, has 1 handler(s) for this service...
Service "orclXDB.localdomain.com" has 2 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
  Instance "orcl2", status READY, has 1 handler(s) for this service...
The command completed successfully
[grid@rac2 bin]$

Relocate Voting disk to +DATA1 to different disks

Step1: Verify voting disk location using below command 

[root@rac1 bin]#
[root@rac1 bin]#
[root@rac1 bin]# ./crsctl   query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   10ee09a0a8a44fe5bf6184e1f3abdc06 (ORCL:DISK1) [DATA1]
Located 1 voting disk(s).
[root@rac1 bin]#
[root@rac1 bin]#


Step2:-  Now move voting disk  to +DATA2 using below command 

[root@rac1 bin]# ./crsctl  replace votedisk +DATA2
Successful addition of voting disk 2edb42aeb93d4fa7bfd06327ae391776.
Successful deletion of voting disk 10ee09a0a8a44fe5bf6184e1f3abdc06.
Successfully replaced voting disk group with +DATA2.
CRS-4266: Voting file(s) successfully replaced
[root@rac1 bin]#
[root@rac1 bin]#

Step3:- verify voting disk location

[root@rac1 bin]# ./crsctl   query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   2edb42aeb93d4fa7bfd06327ae391776 (ORCL:DISK2) [DATA2]
Located 1 voting disk(s).
[root@rac1 bin]#


Note : make sure asm_diskstring configure properly  before relocation

Relocate OCR from +DATA1 to different disks

Task:-
 Relocate OCR from +DATA1 to +DATA2

Requirement:-

Need +DATA2 disk group 


Step 1: Check current OCR location 


[root@rac1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3064
         Available space (kbytes) :     259056
         ID                       :  918475055
         Device/File Name         :     +DATA1
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded


         Logical corruption check succeeded

[root@rac1 bin]#
[root@rac1 bin]# pwd
/u01/app/11.2.0/grid_1/bin
[root@rac1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3064
         Available space (kbytes) :     259056
         ID                       :  918475055
         Device/File Name         :     +DATA1
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

Step 2: Verify asm diskgroup for relocate 

[root@rac1 bin]# oracleasm listdisks
DISK1
DISK2
DISK3

Step 3:-  Now add +DATA2 as new OCR location and verify using below command  
[root@rac1 bin]#
[root@rac1 bin]# ./ocrconfig -add +DATA2
[root@rac1 bin]#
[root@rac1 bin]#
[root@rac1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3064
         Available space (kbytes) :     259056
         ID                       :  918475055
         Device/File Name         :     +DATA1
                                    Device/File integrity check succeeded
         Device/File Name         :     +DATA2
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

Step4:- Now delete old location +DATA1 using below command  and verify

[root@rac1 bin]# ./ocrconfig -delete +DATA1
[root@rac1 bin]#
[root@rac1 bin]#
[root@rac1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3064
         Available space (kbytes) :     259056
         ID                       :  918475055
         Device/File Name         :     +DATA2
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@rac1 bin]#