Applying PSU patch(April 2015) 11.2.0.4.2 Two Node RAC

Grid patch : p18139609_112040_Linux-x86-64.zip

Oracle Grid Infrastructure Patch Set Update 11.2.0.4.2 (Includes Database PSU 11.2.0.4.2) - 18139609

The GI PSU patch includes updates for both the Clusterware home and Database home that can be applied in a rolling fashion.

Step 1 :Take Backup of Grid Home on Both home(using tar command)
Step 2 : Download latest Opatch-6880880
Step 3 : Create OCM (Oracle Configuration Manager) response file
Step 4 : Validation of Oracle Inventory
Step 5 : Download PSU patch Oracle Grid Infrastructure Patch Set Update 11.2.0.4.2 (Includes Database PSU 11.2.0.4.2) - 18139609
(Read carefully README.html file )
Step 6 : One-off Patch Conflict Detection and Resolution
Step 7 : Stop EM Agent Processes Prior to Patching and Prior to Rolling Back the Patch
Step 8 : Patch Installation
Step 9 : Repeat the same steps on second node
Step 10: Patch Post-Installation
Step 11 : Loading Modified SQL Files into the Database

Grid & Database patch:
=================

[root@rac1 OPatch]# ./opatch version
OPatch Version: 11.2.0.3.4

OPatch succeeded.
[root@rac1 OPatch]#


Now we download new OPatch

[root@rac1 OPatch]# ./opatch version
OPatch Version: 11.2.0.3.15

OPatch succeeded.
[root@rac1 OPatch]#






*first check optach version
* create OCM (oracle configuration manager) file

 export ORACLE_HOME=<my_oracle_home_path>
 $ORACLE_HOME/OPatch/ocm/bin/emocmrsp  -no_banner -output <specify_the_location>/file.rsp


 How to Create an OCM Response file to Apply a Patch in Silent Mode - opatch silent (Doc ID 966023.1)


opatch lsinventory -detail -oh <ORACLE_HOME>

opatch lsinventory -detail -oh /u01/app/11.2.0/grid_1



[grid@rac1 OPatch]$ ./opatch lsinventory -detail -oh /u01/app/11.2.0/grid_1
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/11.2.0/grid_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0/grid_1/oraInst.loc
OPatch version    : 11.2.0.3.15
OUI version       : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid_1/cfgtoollogs/opatch/opatch2017-02-04_17-54-41PM_1.log

Lsinventory Output file location : /u01/app/11.2.0/grid_1/cfgtoollogs/opatch/lsinv/lsinventory2017-02-04_17-54-41PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac1.localdomain.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 products installed in this Oracle Home.


Installed Products (89):

Agent Required Support Files                                         10.2.0.4.5
Assistant Common Files                                               11.2.0.4.0
Automatic Storage Management Assistant                               11.2.0.4.0
Bali Share                                                           1.1.18.0.0
Buildtools Common Files                                              11.2.0.4.0
Character Set Migration Utility                                      11.2.0.4.0
Cluster Ready Services Files                                         11.2.0.4.0
Cluster Verification Utility Common Files                            11.2.0.4.0
Cluster Verification Utility Files                                   11.2.0.4.0
Database SQL Scripts                                                 11.2.0.4.0
Deinstallation Tool                                                  11.2.0.4.0
Enterprise Manager Common Core Files                                 10.2.0.4.5
Enterprise Manager Common Files                                      10.2.0.4.5
Enterprise Manager plugin Common Files                               11.2.0.4.0
Expat libraries                                                       2.0.1.0.1
HAS Common Files                                                     11.2.0.4.0
HAS Files for DB                                                     11.2.0.4.0
Installation Common Files                                            11.2.0.4.0
Installation Plugin Files                                            11.2.0.4.0
Installer SDK Component                                              11.2.0.4.0
Java Development Kit                                                1.5.0.51.10
LDAP Required Support Files                                          11.2.0.4.0
OLAP SQL Scripts                                                     11.2.0.4.0
Oracle Advanced Security                                             11.2.0.4.0
Oracle Clusterware RDBMS Files                                       11.2.0.4.0
Oracle Configuration Manager Deconfiguration                         10.3.1.0.0
Oracle Containers for Java                                           11.2.0.4.0
Oracle Core Required Support Files                                   11.2.0.4.0
Oracle Database 11g                                                  11.2.0.4.0
Oracle Database 11g Multimedia Files                                 11.2.0.4.0
Oracle Database Deconfiguration                                      11.2.0.4.0
Oracle Database User Interface                                       2.2.13.0.0
Oracle Database Utilities                                            11.2.0.4.0
Oracle DBCA Deconfiguration                                          11.2.0.4.0
Oracle Extended Windowing Toolkit                                    3.4.47.0.0
Oracle Globalization Support                                         11.2.0.4.0
Oracle Globalization Support                                         11.2.0.4.0
Oracle Grid Infrastructure 11g                                       11.2.0.4.0
Oracle Help For Java                                                  4.2.9.0.0
Oracle Ice Browser                                                    5.2.3.6.0
Oracle Internet Directory Client                                     11.2.0.4.0
Oracle Java Client                                                   11.2.0.4.0
Oracle JDBC/OCI Instant Client                                       11.2.0.4.0
Oracle JDBC/THIN Interfaces                                          11.2.0.4.0
Oracle JFC Extended Windowing Toolkit                                4.2.36.0.0
Oracle JVM                                                           11.2.0.4.0
Oracle LDAP administration                                           11.2.0.4.0
Oracle Locale Builder                                                11.2.0.4.0
Oracle Multimedia                                                    11.2.0.4.0
Oracle Multimedia Client Option                                      11.2.0.4.0
Oracle Multimedia Java Advanced Imaging                              11.2.0.4.0
Oracle Multimedia Locator                                            11.2.0.4.0
Oracle Multimedia Locator RDBMS Files                                11.2.0.4.0
Oracle Net                                                           11.2.0.4.0
Oracle Net Listener                                                  11.2.0.4.0
Oracle Net Required Support Files                                    11.2.0.4.0
Oracle Netca Client                                                  11.2.0.4.0
Oracle Notification Service                                          11.2.0.3.0
Oracle Notification Service (eONS)                                   11.2.0.4.0
Oracle One-Off Patch Installer                                       11.2.0.3.4
Oracle Quality of Service Management (Client)                        11.2.0.4.0
Oracle Quality of Service Management (Server)                        11.2.0.4.0
Oracle RAC Deconfiguration                                           11.2.0.4.0
Oracle RAC Required Support Files-HAS                                11.2.0.4.0
Oracle Recovery Manager                                              11.2.0.4.0
Oracle Security Developer Tools                                      11.2.0.4.0
Oracle Text Required Support Files                                   11.2.0.4.0
Oracle Universal Installer                                           11.2.0.4.0
Oracle USM Deconfiguration                                           11.2.0.4.0
Oracle Wallet Manager                                                11.2.0.4.0
Parser Generator Required Support Files                              11.2.0.4.0
Perl Interpreter                                                     5.10.0.0.2
Perl Modules                                                         5.10.0.0.1
PL/SQL                                                               11.2.0.4.0
PL/SQL Embedded Gateway                                              11.2.0.4.0
Platform Required Support Files                                      11.2.0.4.0
Precompiler Required Support Files                                   11.2.0.4.0
RDBMS Required Support Files                                         11.2.0.4.0
RDBMS Required Support Files for Instant Client                      11.2.0.4.0
RDBMS Required Support Files Runtime                                 11.2.0.4.0
Required Support Files                                               11.2.0.4.0
Secure Socket Layer                                                  11.2.0.4.0
SQL*Plus                                                             11.2.0.4.0
SQL*Plus Files for Instant Client                                    11.2.0.4.0
SQL*Plus Required Support Files                                      11.2.0.4.0
SSL Required Support Files for InstantClient                         11.2.0.4.0
Universal Storage Manager Files                                      11.2.0.4.0
XDK Required Support Files                                           11.2.0.4.0
XML Parser for Java                                                  11.2.0.4.0
There are 89 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


Rac system comprising of multiple nodes
  Local node = rac1
  Remote node = rac2

--------------------------------------------------------------------------------

OPatch succeeded.
[grid@rac1 OPatch]$



check patch conflicts


./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./18139609



[grid@rac1 oracle_software]$ /u01/app/11.2.0/grid_1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./18139609
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2017, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /u01/app/11.2.0/grid_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0/grid_1/oraInst.loc
OPatch version    : 11.2.0.3.15
OUI version       : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid_1/cfgtoollogs/opatch/opatch2017-02-04_18-28-00PM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.
[grid@rac1 oracle_software]$



[oracle@rac1 ~]$ emctl status dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.4.0
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
https://rac1.localdomain.com:1158/em/console/aboutApplication
Oracle Enterprise Manager 11g is not running.
------------------------------------------------------------------
Logs are generated in directory /u01/app/oracle/product/11.2.0/dbhome_1/rac1_orcl/sysman/log
[oracle@rac1 ~]$

  
 example:
 ===========
 cd 12345
% opatch apply -silent -ocmrf <path_to_response_file>




[root@rac1 oracle_software]#
[root@rac1 oracle_software]#
[root@rac1 oracle_software]# echo $GRID_HOME
/u01/app/11.2.0/grid_1/
[root@rac1 oracle_software]#
[root@rac1 oracle_software]# echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0/dbhome_1/
[root@rac1 oracle_software]#
[root@rac1 oracle_software]#
[[root@rac1 oracle_software]#
[root@rac1 oracle_software]#
[root@rac1 oracle_software]# chmod -R 775 /u01/app/11.2.0/grid_1/
[root@rac1 oracle_software]#
[root@rac1 oracle_software]#
[root@rac1 oracle_software]# /u01/app/11.2.0/grid_1/OPatch/opatch auto /u01/oracle_software/18139609 -ocmrf /u01/oracle_software/orcl_grid.rsp
Executing /u01/app/11.2.0/grid_1/perl/bin/perl /u01/app/11.2.0/grid_1/OPatch/crs/patch11203.pl -patchdir /u01/oracle_software -patchn 18139609 -ocmrf /u01/oracle_software/orcl_grid.rsp -paramfile /u01/app/11.2.0/grid_1/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.0/grid_1/cfgtoollogs/opatchauto2017-02-04_19-26-58.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.0/grid_1/cfgtoollogs/opatchauto2017-02-04_19-26-58.report.log

2017-02-04 19:26:58: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.0/grid_1/crs/install/crsconfig_params


Stopping RAC /u01/app/oracle/product/11.2.0/dbhome_1 ...
Stopped RAC /u01/app/oracle/product/11.2.0/dbhome_1 successfully

patch /u01/oracle_software/18139609/18031668  apply successful for home  /u01/app/oracle/product/11.2.0/dbhome_1
patch /u01/oracle_software/18139609/18031740/custom/server/18031740  apply successful for home  /u01/app/oracle/product/11.2.0/dbhome_1

Stopping CRS...
Stopped CRS successfully

patch /u01/oracle_software/18139609/18031668  apply successful for home  /u01/app/11.2.0/grid_1
patch /u01/oracle_software/18139609/18031740  apply successful for home  /u01/app/11.2.0/grid_1
patch /u01/oracle_software/18139609/18031731  apply successful for home  /u01/app/11.2.0/grid_1

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

Starting RAC /u01/app/oracle/product/11.2.0/dbhome_1 ...
Started RAC /u01/app/oracle/product/11.2.0/dbhome_1 successfully

opatch auto succeeded.
[root@rac1 oracle_software]#
[root@rac1 oracle_software]#






SQL>  select INSTANCE_NUMBER,INSTANCE_NAME,HOST_NAME,STARTUP_TIME,STATUS from gv$instance;

INSTANCE_NUMBER INSTANCE_NAME    HOST_NAME                                                        STARTUP_TIME         STATUS
--------------- ---------------- ---------------------------------------------------------------- -------------------- ------------
              2 orcl2            rac2.localdomain.com                                             04-FEB-2017 04:08:10 OPEN
              1 orcl1            rac1.localdomain.com                                             04-FEB-2017 19:54:25 OPEN

SQL>







On Node2=============



[root@rac2 oracle_software]# cp -r OPatch/ /u01/app/11.2.0/grid_1/
[root@rac2 oracle_software]#
[root@rac2 oracle_software]#
[root@rac2 oracle_software]# cd /u01/app/11.2.0/grid_1/
[root@rac2 grid_1]#
[root@rac2 grid_1]# chmod -R 777 OPatch
[root@rac2 grid_1]#
[root@rac2 grid_1]#
[root@rac2 grid_1]#
[root@rac2 grid_1]#
[root@rac2 grid_1]# chmod -R 777 /u01/oracle_software/
[root@rac2 grid_1]#
[root@rac2 grid_1]#
[root@rac2 grid_1]#
[root@rac2 grid_1]#
[root@rac2 grid_1]# export GRID_HOME=/u01/app/11.2.0/grid_1/
[root@rac2 grid_1]#
[root@rac2 grid_1]# export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1/
[root@rac2 grid_1]#
[root@rac2 grid_1]#
[root@rac2 grid_1]# echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0/dbhome_1/
[root@rac2 grid_1]#
[root@rac2 grid_1]# echo $GRID_HOME
/u01/app/11.2.0/grid_1/
[root@rac2 grid_1]#
[root@rac2 grid_1]#
[root@rac2 grid_1]# chmod -R 775 /u01/app/11.2.0/grid_1/
[root@rac2 grid_1]#
[root@rac2 grid_1]#
[root@rac2 oracle_software]# chmod -R 777 orcl_grid.rsp
[root@rac2 oracle_software]#
[root@rac2 oracle_software]# ls -lrt
total 615524
-rwxrwxrwx  1 grid oinstall  98352086 Feb  4 21:21 p6880880_112000_Linux-x86-64.zip
-rwxrwxrwx  1 grid oinstall     23977 Feb  4 21:21 PatchSearch.xml
-rwxrwxrwx  1 grid oinstall 531900097 Feb  4 21:22 p18139609_112040_Linux-x86-64.zip
drwxrwxrwx 15 grid oinstall      4096 Feb  4 21:22 OPatch
drwxrwxrwx  5 grid oinstall      4096 Feb  4 21:24 18139609
-rwxrwxrwx  1 grid oinstall      1067 Feb  4 21:27 orcl_grid.rsp
[root@rac2 oracle_software]#
[root@rac2 oracle_software]# /u01/app/11.2.0/grid_1/OPatch/opatch auto /u01/oracle_software/18139609 -ocmrf /u01/oracle_software/orcl_grid.rsp
Executing /u01/app/11.2.0/grid_1/perl/bin/perl /u01/app/11.2.0/grid_1/OPatch/crs/patch11203.pl -patchdir /u01/oracle_software -patchn 18139609 -ocmrf /u01/oracle_software/orcl_grid.rsp -paramfile /u01/app/11.2.0/grid_1/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.0/grid_1/cfgtoollogs/opatchauto2017-02-04_22-00-46.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.0/grid_1/cfgtoollogs/opatchauto2017-02-04_22-00-46.report.log

2017-02-04 22:00:46: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.0/grid_1/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.0/dbhome_1 ...
Stopped RAC /u01/app/oracle/product/11.2.0/dbhome_1 successfully

patch /u01/oracle_software/18139609/18031668  apply successful for home  /u01/app/oracle/product/11.2.0/dbhome_1
patch /u01/oracle_software/18139609/18031740/custom/server/18031740  apply successful for home  /u01/app/oracle/product/11.2.0/dbhome_1

Stopping CRS...
Stopped CRS successfully

patch /u01/oracle_software/18139609/18031668  apply successful for home  /u01/app/11.2.0/grid_1
patch /u01/oracle_software/18139609/18031740  apply successful for home  /u01/app/11.2.0/grid_1
patch /u01/oracle_software/18139609/18031731  apply successful for home  /u01/app/11.2.0/grid_1

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

Starting RAC /u01/app/oracle/product/11.2.0/dbhome_1 ...
Started RAC /u01/app/oracle/product/11.2.0/dbhome_1 successfully

opatch auto succeeded.
[root@rac2 oracle_software]#


select INSTANCE_NUMBER,INSTANCE_NAME,HOST_NAME,STARTUP_TIME,STATUS from gv$instance;

INSTANCE_NUMBER INSTANCE_NAME    HOST_NAME                                                        STARTUP_TIME         STATUS
--------------- ---------------- ---------------------------------------------------------------- -------------------- ------------
              1 orcl1            rac1.localdomain.com                                             04-FEB-2017 19:54:25 OPEN
              2 orcl2            rac2.localdomain.com                                             04-FEB-2017 22:29:58 OPEN


 cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @catbundle.sql psu apply
SQL> QUIT



cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @utlrp.sql

 SQL> select * from dba_registry_history;

ACTION_TIME         ACTION                NAMESPACE  VERSION          ID BUNDLE_SERIES            COMMENTS
-------------------- ------------------------------ ---------- ---------- ---------- ------------------------------ --------------------
24-AUG-13 12.03.45.1 APPLY                SERVER     11.2.0.4        0 PSU                Patchset 11.2.0.2.0
19862 PM

20-SEP-16 01.55.30.3 APPLY                SERVER     11.2.0.4        0 PSU                Patchset 11.2.0.2.0
72018 PM

05-FEB-17 01.34.14.8 APPLY                SERVER     11.2.0.4        2 PSU                PSU 11.2.0.4.2
86008 AM


SQL>



Error face during patching
=======================


2017-02-04 18:35:47: Oracle Interim Patch Installer version 11.2.0.3.15
 Copyright (c) 2017, Oracle Corporation.  All rights reserved.


 Oracle Home       : /u01/app/11.2.0/grid_1
 Central Inventory : /u01/app/oraInventory
    from           : /u01/app/11.2.0/grid_1/oraInst.loc
 OPatch version    : 11.2.0.3.15
 OUI version       : 11.2.0.4.0
 Log file location : /u01/app/11.2.0/grid_1/cfgtoollogs/opatch/opatch2017-02-04_18-35-46PM_1.log

 OPatch failed to lock and get an Inventory Session for the given Oracle Home /u01/app/11.2.0/grid_1
 Possible causes are:
    No read or write permission to $ORACLE_HOME, cannot create $ORACLE_HOME/.patch_storage
    No read or write permission to $ORACLE_HOME/.patch_storage
    OPatch cannot proceed further because system will not be able to backup files, or read the backup area for rollback/restore.
 OPatchSession cannot load inventory for the given Oracle Home /u01/app/11.2.0/grid_1. Possible causes are:
    No read or write permission to ORACLE_HOME/.patch_storage
    Central Inventory is locked by another OUI instance
    No read permission to Central Inventory
    The lock file exists in ORACLE_HOME/.patch_storage
    The Oracle Home does not exist in Central Inventory

 UtilSession failed: IPMRWServices::verifyPatchStorageDirectory() cannot read or write to /u01/app/11.2.0/grid_1/.patch_storage

 OPatch failed with error code 73

2017-02-04 18:35:47: Status of opatch version check  for /u01/app/11.2.0/grid_1 is 0
2017-02-04 18:35:47: Opatch version check failed for oracle home  /u01/app/11.2.0/grid_1
2017-02-04 18:35:47: Opatch version  check failed
2017-02-04 18:35:47: ERROR: update the opatch version for the failed homes and retry

solution : give permission to grid home



[root@rac1 oracle_software]#
[root@rac1 oracle_software]# chmod -R 775 /u01/app/11.2.0/grid_1/
[root@rac1 oracle_software]#

===================================================

[oracle@rac1 OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
OPatch version    : 11.2.0.3.15
OUI version       : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2017-02-05_02-31-06AM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2017-02-05_02-31-06AM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac1.localdomain.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.4.0
There are 1 products installed in this Oracle Home.


Interim patches (2) :

Patch  18031740     : applied on Sat Feb 04 19:35:58 IST 2017
Unique Patch ID:  17253722
Patch description:  "OCW Patch Set Update : 11.2.0.4.2 (18031740)"
   Created on 19 Mar 2014, 09:06:31 hrs PST8PDT
   Bugs fixed:
     18328800, 16613232, 14525998, 14693336, 17273020, 17238586, 12928658
     17391726, 17750548, 17387214, 18229842, 17172091, 17001914, 16346413
     17155238, 17551223, 15832129, 17305100, 17483479, 17039197, 16317771
     17292250, 17500165, 16206997, 17481314, 16876500, 17405302, 17065496
     18343490, 13991403

Patch  18031668     : applied on Sat Feb 04 19:32:54 IST 2017
Unique Patch ID:  17255543
Patch description:  "Database Patch Set Update : 11.2.0.4.2 (18031668)"
   Created on 20 Feb 2014, 05:15:58 hrs PST8PDT
Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
   Bugs fixed:
     17332800, 17284817, 17811456, 17752121, 14338435, 13944971, 18031668
     17394950, 17437634, 16943711, 17186905, 17080436, 16731148, 17027426
     18094246, 13866822, 16863422, 16043574, 17783588, 17811429, 17082359
     17385178, 17443671, 17610798, 17390431, 16875449, 16069901, 17478514
     17296856, 17811447, 16613964, 14010183, 18180390, 16850630, 16785708
     17865671, 17612828, 10136473, 17088068, 17016369, 17325413, 17323222
     16180763, 17811438, 16912439, 17040764, 13364795, 16228604, 17545847
     17346671, 16472716, 16285691, 17726838, 17614227, 17393683, 17551709
     17341326, 17546973, 17050888, 17344412, 17721717, 17546761, 17375354
     16618694, 18203835, 17446237, 18203838, 18203837, 17313525, 12905058
     14852021, 17288409, 16956380, 18139690, 16399083, 17716305, 16837842
     17622427, 14133975, 17501491, 17761775, 17389192, 17468141, 16450169
     17265217, 17441661, 17071721, 17465741, 17235750, 16721594, 17205719
     16220077, 17602269, 16929165, 17239687



Rac system comprising of multiple nodes
  Local node = rac1
  Remote node = rac2

--------------------------------------------------------------------------------

OPatch succeeded.
[oracle@rac1 OPatch]$





[grid@rac1 OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/11.2.0/grid_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0/grid_1/oraInst.loc
OPatch version    : 11.2.0.3.15
OUI version       : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid_1/cfgtoollogs/opatch/opatch2017-02-05_02-32-34AM_1.log

Lsinventory Output file location : /u01/app/11.2.0/grid_1/cfgtoollogs/opatch/lsinv/lsinventory2017-02-05_02-32-34AM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac1.localdomain.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 products installed in this Oracle Home.


Interim patches (3) :

Patch  18031731     : applied on Sat Feb 04 19:50:43 IST 2017
Unique Patch ID:  17253722
Patch description:  "ACFS Patch Set Update : 11.2.0.4.2 (18031731)"
   Created on 17 Mar 2014, 06:35:22 hrs PST8PDT
   Bugs fixed:
     17172303, 17203009, 17376318, 17363999, 17721778, 17611362, 17164243
     17428148, 17070158, 17699423, 17475946

Patch  18031740     : applied on Sat Feb 04 19:47:12 IST 2017
Unique Patch ID:  17253722
Patch description:  "OCW Patch Set Update : 11.2.0.4.2 (18031740)"
   Created on 19 Mar 2014, 09:06:37 hrs PST8PDT
   Bugs fixed:
     18328800, 16613232, 14525998, 14693336, 17273020, 17238586, 12928658
     17391726, 17750548, 17387214, 18229842, 17172091, 17001914, 16346413
     17155238, 17551223, 15832129, 17305100, 17483479, 17039197, 16317771
     17292250, 17500165, 16206997, 17481314, 16876500, 17405302, 17065496
     18343490, 13991403

Patch  18031668     : applied on Sat Feb 04 19:44:09 IST 2017
Unique Patch ID:  17255543
Patch description:  "Database Patch Set Update : 11.2.0.4.2 (18031668)"
   Created on 20 Feb 2014, 05:15:58 hrs PST8PDT
Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
   Bugs fixed:
     17332800, 17284817, 17811456, 17752121, 14338435, 13944971, 18031668
     17394950, 17437634, 16943711, 17186905, 17080436, 16731148, 17027426
     18094246, 13866822, 16863422, 16043574, 17783588, 17811429, 17082359
     17385178, 17443671, 17610798, 17390431, 16875449, 16069901, 17478514
     17296856, 17811447, 16613964, 14010183, 18180390, 16850630, 16785708
     17865671, 17612828, 10136473, 17088068, 17016369, 17325413, 17323222
     16180763, 17811438, 16912439, 17040764, 13364795, 16228604, 17545847
     17346671, 16472716, 16285691, 17726838, 17614227, 17393683, 17551709
     17341326, 17546973, 17050888, 17344412, 17721717, 17546761, 17375354
     16618694, 18203835, 17446237, 18203838, 18203837, 17313525, 12905058
     14852021, 17288409, 16956380, 18139690, 16399083, 17716305, 16837842
     17622427, 14133975, 17501491, 17761775, 17389192, 17468141, 16450169
     17265217, 17441661, 17071721, 17465741, 17235750, 16721594, 17205719
     16220077, 17602269, 16929165, 17239687



Rac system comprising of multiple nodes
  Local node = rac1
  Remote node = rac2

--------------------------------------------------------------------------------

OPatch succeeded.
[grid@rac1 OPatch]$

Create an OCM(Oracle Configuration Manager) Response file to Apply a Patch in RAC

Step1 : -
          Download  and install latest OPatch

Patch 6880880: OPatch patch of version 11.2.0.3.15 
for Oracle software releases 11.2.0.x (AUG 2016)

Step 2:
          create  response  file for  patch


                  cd /u01/app/11.2.0/grid_1/OPatch/ocm/bin/

                    ./emocmrsp -no_banner -output /u01/oracle_software/orcl_grid.rsp


Give permission  777 to orcl_grid.rsp file


How to use :

[root@rac2 oracle_software]# /u01/app/11.2.0/grid_1/OPatch/opatch auto /u01/oracle_software/18139609 -ocmrf /u01/oracle_software/orcl_grid.rsp


How to Create an OCM Response file to Apply a 
Patch in Silent Mode - opatch silent (Doc ID 966023.1)

Enable and Disable oracle feature using chopt command : RAT disable using chopt command

RAT disable in oracle RAC using chopt command 
==========================================================

Check  RAT status in database using below query

SQL>  select * from gv$option where parameter='Real Application Testing';

   INST_ID PARAMETER                                          VALUE
---------- -------------------------------------------------- ----------------------------------------------------------------
         1 Real Application Testing                           TRUE
         2 Real Application Testing                           TRUE

SQL>
SQL>


Check database  and listener status using below command
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status database -d orcl
Instance orcl1 is running on node rac1
Instance orcl2 is running on node rac2
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): rac2,rac1
[oracle@rac1 ~]$
[oracle@rac1 ~]$

Stop listener and database services on all node in case RAC

[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl stop listener
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is not running
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl stop database -d orcl
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status database -d orcl
Instance orcl1 is not running on node rac1
Instance orcl2 is not running on node rac2
[oracle@rac1 ~]$
[oracle@rac1 ~]$

Check database and listener status on second node : RAC2

[oracle@rac2 ~]$
[oracle@rac2 ~]$
[oracle@rac2 ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is not running
[oracle@rac2 ~]$
[oracle@rac2 ~]$ srvctl status database -d orcl
Instance orcl1 is not running on node rac1
Instance orcl2 is not running on node rac2
[oracle@rac2 ~]$
[oracle@rac2 ~]$

Go to $ORACLE_HOME/bin location using cd command  on node rac1


[oracle@rac1 bin]$ pwd
/u01/app/oracle/product/11.2.0/dbhome_1/bin
[oracle@rac1 bin]$
[oracle@rac1 bin]$ ls -lrt *chopt*
-rwxr-xr-x 1 oracle oinstall  234 Sep 20 13:07 chopt
-rw-r--r-- 1 oracle oinstall 6384 Sep 20 13:07 chopt.pl
-rw-r--r-- 1 oracle oinstall 3780 Sep 20 13:07 chopt.ini
[oracle@rac1 bin]$
[oracle@rac1 bin]$

Disable RAT on node rac1 using below command

[oracle@rac1 bin]$ ./chopt disable rat

Writing to /u01/app/oracle/product/11.2.0/dbhome_1/install/disable_rat.log...
/usr/bin/make -f /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/lib/ins_rdbms.mk rat_off ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
/usr/bin/make -f /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/lib/ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1

[oracle@rac1 bin]$

Same thing do on second node rac2
[oracle@rac2 bin]$ pwd
/u01/app/oracle/product/11.2.0/dbhome_1/bin
[oracle@rac2 bin]$
[oracle@rac2 bin]$ ls -lrt *chopt*
-rwxr-xr-x 1 oracle oinstall  234 Sep 20 13:18 chopt
-rw-r--r-- 1 oracle oinstall 3780 Sep 20 13:18 chopt.ini
-rw-r--r-- 1 oracle oinstall 6384 Sep 20 13:18 chopt.pl
[oracle@rac2 bin]$
[oracle@rac2 bin]$

[oracle@rac2 bin]$ pwd
/u01/app/oracle/product/11.2.0/dbhome_1/bin
[oracle@rac2 bin]$
[oracle@rac2 bin]$ ls -lrt *chopt*
-rwxr-xr-x 1 oracle oinstall  234 Sep 20 13:18 chopt
-rw-r--r-- 1 oracle oinstall 3780 Sep 20 13:18 chopt.ini
-rw-r--r-- 1 oracle oinstall 6384 Sep 20 13:18 chopt.pl
[oracle@rac2 bin]$
[oracle@rac2 bin]$ ./chopt disable rat

Writing to /u01/app/oracle/product/11.2.0/dbhome_1/install/disable_rat.log...
/usr/bin/make -f /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/lib/ins_rdbms.mk rat_off ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
/usr/bin/make -f /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/lib/ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1

[oracle@rac2 bin]$


start database and listener services  on all node  RAC1,RAC2  and verify

[oracle@rac1 bin]$
[oracle@rac1 bin]$ srvctl start database -d orcl
[oracle@rac1 bin]$
[oracle@rac1 bin]$
[oracle@rac1 bin]$ srvctl status database -d orcl
Instance orcl1 is running on node rac1
Instance orcl2 is running on node rac2
[oracle@rac1 bin]$
[oracle@rac1 bin]$
[oracle@rac1 bin]$ srvctl start listener
[oracle@rac1 bin]$
[oracle@rac1 bin]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): rac2,rac1
[oracle@rac1 bin]$
[oracle@rac1 bin]$


Then verify using below command


SQL>
SQL> select * from gv$option where parameter='Real Application Testing';

   INST_ID PARAMETER                                          VALUE
---------- -------------------------------------------------- ----------------------------------------------------------------
         1 Real Application Testing                           FALSE
         2 Real Application Testing                           FALSE

SQL>



How to Check and Enable/Disable Oracle Binary Options (Doc ID 948061.1)

Note: In case of RAC environment, the chopt enable / disable <option> command has to be executed on all RAC nodes.

[oracle@rac1 bin]$ chopt -h

usage:

chopt <enable|disable> <option>

options:
                  dm = Oracle Data Mining RDBMS Files
                  dv = Oracle Database Vault option
                lbac = Oracle Label Security
                olap = Oracle OLAP
        partitioning = Oracle Partitioning
                 rat = Oracle Real Application Testing

e.g. chopt enable rat


[oracle@rac1 bin]$

Start and Stop CRS (Cluster Ready Services) in RAC

 This command stop whole services (CRS+HA) on current node only (rac1)

[root@rac1 bin]# ./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.DATA1.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'rac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac1'
CRS-2677: Stop of 'ora.cvu' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2676: Start of 'ora.cvu' on 'rac2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rac2'
CRS-2677: Stop of 'ora.orcl.db' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATA2.dg' on 'rac1'
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
CRS-2676: Start of 'ora.scan3.vip' on 'rac2' succeeded
CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rac2' succeeded
CRS-2677: Stop of 'ora.DATA2.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.DATA1.dg' on 'rac1' succeeded
CRS-2679: Attempting to clean 'ora.DATA1.dg' on 'rac1'
CRS-2681: Clean of 'ora.DATA1.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1'
CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1 bin]#
[root@rac1 bin]#
[root@rac1 bin]# ps -ef|grep d.bin
root      2206 22673  0 02:58 pts/1    00:00:00 grep d.bin
[root@rac1 bin]#
[root@rac1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 bin]#

wait for some time all services will come  online 


[root@rac1 bin]#  ps -ef|grep d.bin
root      2582     1  1 03:01 ?        00:00:02 /u01/app/11.2.0/grid_1/bin/ohasd.bin reboot
grid      2733     1  0 03:01 ?        00:00:00 /u01/app/11.2.0/grid_1/bin/mdnsd.bin
grid      2744     1  0 03:01 ?        00:00:00 /u01/app/11.2.0/grid_1/bin/gpnpd.bin
grid      2759     1  0 03:01 ?        00:00:01 /u01/app/11.2.0/grid_1/bin/gipcd.bin
root      2775     1  1 03:01 ?        00:00:01 /u01/app/11.2.0/grid_1/bin/osysmond.bin
grid      2816     1  1 03:01 ?        00:00:01 /u01/app/11.2.0/grid_1/bin/ocssd.bin
root      2922     1  0 03:02 ?        00:00:00 /u01/app/11.2.0/grid_1/bin/octssd.bin reboot
grid      2954     1  0 03:02 ?        00:00:00 /u01/app/11.2.0/grid_1/bin/evmd.bin
root      3030     1  1 03:02 ?        00:00:01 /u01/app/11.2.0/grid_1/bin/crsd.bin reboot
root      3620 22673  0 03:03 pts/1    00:00:00 grep d.bin
[root@rac1 bin]#



[root@rac1 bin]# ./crsctl check cluster -all
**************************************************************
rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@rac1 bin]#

[root@rac1 bin]# ./crsctl stat res -init -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       rac1                     Started
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       rac1
ora.crf
      1        ONLINE  ONLINE       rac1
ora.crsd
      1        ONLINE  ONLINE       rac1
ora.cssd
      1        ONLINE  ONLINE       rac1
ora.cssdmonitor
      1        ONLINE  ONLINE       rac1
ora.ctssd
      1        ONLINE  ONLINE       rac1                     ACTIVE:0
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       rac1
ora.gipcd
      1        ONLINE  ONLINE       rac1
ora.gpnpd
      1        ONLINE  ONLINE       rac1
ora.mdnsd
      1        ONLINE  ONLINE       rac1
[root@rac1 bin]#
[root@rac1 bin]#

[root@rac1 bin]#  ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA1.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.DATA2.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.asm
               ONLINE  ONLINE       rac1                     Started
               ONLINE  ONLINE       rac2                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
               OFFLINE OFFLINE      rac2
ora.net1.network
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.ons
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac2
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac2
ora.cvu
      1        ONLINE  ONLINE       rac2
ora.oc4j
      1        ONLINE  ONLINE       rac2
ora.orcl.db
      1        ONLINE  ONLINE       rac1                     Open
      2        ONLINE  ONLINE       rac2                     Open
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.rac2.vip
      1        ONLINE  ONLINE       rac2
ora.scan1.vip
      1        ONLINE  ONLINE       rac1
ora.scan2.vip
      1        ONLINE  ONLINE       rac2
ora.scan3.vip
      1        ONLINE  ONLINE       rac2
[root@rac1 bin]#


This command stop and start  cluster services local node only .
Need to do on each node to stop whole cluster-ware 


http://www.freelists.org/post/oracle-l/crsctl-stop-has-vs-crsctl-stop-crs,2