ORAchk #1 : Installing and use the ORAchk Health Check Tool

 ORAchk  VERSION: 12.2.0.1.4 - Health Checks for the Oracle Stack (Doc ID 1268927.2)
========================================================================


Download the ORAchk: The latest version of ORAchk is attached to Note 1268927.1 and can be downloaded    "ORAchk stand-alone"





Unzip orachk.zip kit $ unzip orachk.zip
===============================================

[root@rac1 orachk]# ls -lrt
total 257500
-rw-r--r-- 1 grid oinstall 263673609 May 16 21:32 orachk_2.zip
[root@rac1 orachk]#
[root@rac1 orachk]# unzip orachk_2.zip
Archive:  orachk_2.zip
   creating: .cgrep/

 
After unzip Give to permission $ chmod 755 orachk
=======================================================

[root@rac1 oracle_software]# cd orachk/
[root@rac1 orachk]#
[root@rac1 orachk]# ls -lrt
total 325364
-rwxrwxrwx 1 root root       6424905 Dec 12 22:33 rules.dat
-rwxrwxrwx 1 root root      49352682 Dec 12 22:33 collections.dat
-rwxrwxrwx 1 root root        109317 Dec 12 23:11 orachk.pyc
drwxrwxrwx 2 root root          4096 Dec 12 23:11 bash
-rwxrwxrwx 1 root root          5665 Dec 12 23:11 orachk.bat
-rwxrwxrwx 1 root root       2958609 Dec 12 23:12 CollectionManager_App.sql
-rwxrwxrwx 1 root root       8233480 Dec 12 23:12 Apex5_CollectionManager_App.sql
-rwxrwxrwx 1 root root         40052 Dec 12 23:12 sample_user_defined_checks.xml
-rwxrwxrwx 1 root root          2888 Dec 12 23:12 user_defined_checks.xsd
drwxrwxrwx 2 root root          4096 Dec 12 23:12 build
-rwxrwxrwx 1 root root         11063 Dec 13 03:01 orachk
drwxrwxrwx 3 root root          4096 Dec 13 03:01 lib
drwxrwxrwx 2 root root          4096 Dec 13 03:01 templates
drwxrwxrwx 2 root root          4096 Dec 13 03:01 exadiscover
-rwxrwxrwx 1 root root           230 Dec 13 03:15 UserGuide.txt
-rwxrwxrwx 1 root root         10105 Dec 13 03:15 readme.txt
-rwxrwxrwx 1 root root       2296616 Dec 13 03:15 ORAchk_Health_Check_Catalog.html
-rwxrwxrwx 1 grid oinstall 263673609 May 16 21:32 orachk_2.zip
drwxrwxrwx 2 root root          4096 May 16 21:47 doc
[root@rac1 orachk]#
[root@rac1 orachk]#



[root@rac1 orachk]#
[root@rac1 orachk]# ./orachk
This version of orachk was released on 12-Dec-2017 and its older than 120 days. No new version of orachk is available in RAT_UPGRADE_LOC. It is highly recommended that you download the latest version of orachk from my oracle support to ensure the highest level of accuracy of the data contained within the report.


Do you want to download latest version from my oracle support? [y/n] [y] n

Do you want to continue running this version? [y/n][y] y
Running older version...


CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/app/11.2.0/grid_1?[y/n][y] y

Checking ssh user equivalency settings on all nodes in cluster for root

Node rac2 is not configured for ssh user equivalency and the orachk uses ssh to execute checks on remote nodes.
Without passwordless ssh orachk  can not run audit checks on the remote nodes.
If necessary due to security policies the orachk can be run on each node using -localonly option.


Do you want to configure SSH for user root on rac2 [y/n][y] y

Enter rac2 root password :
Verifying root password ...

Node rac2 has been configured for ssh user equivalency for root


Searching for running databases . . . . .

.  .
List of running databases registered in OCR

1. orcl
2. None of above

Select databases from list for checking best practices. For multiple databases, select 1 for All or comma separated number like 1,2 etc [1-2][1]. 1
.  .  .  .

Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS

.  .  .  . . . .
.  .  . . . .  .  .  .  .  .  .  .  .  . . . .  .  .  .  .  .  .  .
-------------------------------------------------------------------------------------------------------
                                                 Oracle Stack Status
-------------------------------------------------------------------------------------------------------
  Host Name        CRS Installed   RDBMS Installed     CRS UP     ASM UP   RDBMS UP     DB Instance Name
-------------------------------------------------------------------------------------------------------
       rac1                        Yes           Yes           Yes       Yes       Yes                orcl1
       rac2                        Yes           Yes           Yes       Yes       Yes                orcl2
-------------------------------------------------------------------------------------------------------


Copying plug-ins

. .
.  .  .  .  .  .

*** Checking Best Practice Recommendations ( PASS / WARNING / FAIL ) ***


.

Collections and audit checks log file is
/u01/oracle_software/orachk/orachk_rac1_orcl_051618_215243/log/orachk.log

Starting to run orachk in background on rac2



============================================================
                   Node name - rac1
============================================================

Collecting - ASM Disk Groups
Collecting - ASM Disk I/O stats
Collecting - ASM Diskgroup Attributes
Collecting - ASM disk partnership imbalance
Collecting - ASM diskgroup attributes
Collecting - ASM diskgroup usable free space
Collecting - ASM initialization parameters
Collecting - Active sessions load balance for orcl database
Collecting - Archived Destination Status for orcl database
Collecting - Cluster Interconnect Config for orcl database
Collecting - Database Archive Destinations for orcl database
Collecting - Database Files for orcl database
Collecting - Database Instance Settings for orcl database
Collecting - Database Parameters for orcl database
Collecting - Database Properties for orcl database
Collecting - Database Registry for orcl database
Collecting - Database Sequences for orcl database
Collecting - Database Undocumented Parameters for orcl database
Collecting - Database Undocumented Parameters for orcl database
Collecting - Database Workload Services for orcl database
Collecting - Dataguard Status for orcl database
Collecting - Files not opened by ASM
Collecting - Log Sequence Numbers for orcl database
Collecting - Percentage of asm disk  Imbalance
Collecting - Process for shipping Redo to standby for orcl database
Collecting - RDBMS Feature Usage for orcl database
Collecting - Redo Log information for orcl database
Collecting - Standby redo log creation status before switchover for orcl database
Collecting - /proc/cmdline
Collecting - /proc/modules
Collecting - CPU Information
Collecting - CRS active version
Collecting - CRS oifcfg
Collecting - CRS software version
Collecting - CSS Reboot time
Collecting - CSS disktimout
Collecting - Cluster interconnect (clusterware)
Collecting - Clusterware OCR healthcheck
Collecting - Clusterware Resource Status
Collecting - Disk I/O Scheduler on Linux
Collecting - DiskFree Information
Collecting - DiskMount Information
Collecting - Huge pages configuration
Collecting - Interconnect network card speed
Collecting - Kernel parameters
Collecting - Linux module config.
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - Maximum number of semaphores per semaphore set
Collecting - Memory Information
Collecting - NUMA Configuration
Collecting - Network Interface Configuration
Collecting - Network Performance
Collecting - Network Service Switch
Collecting - OS Packages
Collecting - OS version
Collecting - Operating system release information and kernel version
Collecting - Oracle Executable Attributes
Collecting - Patches for Grid Infrastructure
Collecting - Patches for RDBMS Home
Collecting - Shared memory segments
Collecting - Table of file system defaults
Collecting - Voting disks (clusterware)
Collecting - number of semaphore operations per semop system call
Collecting - ACFS Volumes status
Collecting - Broadcast Requirements for Networks
Collecting - CRS Opatch version
Collecting - CRS user time zone check
Collecting - Custom rc init scripts (rc.local)
Collecting - Disk Information
Collecting - Grid Infastructure user shell limits configuration
Collecting - Interconnect interface config
Collecting - Linux Operating system health check using vmpscan.sh
Collecting - Network interface stats
Collecting - Root user limits
Collecting - Verify no database server kernel out of memory errors
Collecting - root time zone check
Collecting - slabinfo
Collecting - umask setting for GI owner


Data collections completed. Checking best practices on rac1.
------------------------------------------------------------



 INFO =>     Important Automatic Storage Management (ASM) Notes and Technical White Papers
 INFO =>     Oracle Data Pump Best practices.
 FAIL =>     Bash is vulnerable to code injection (CVE-2014-6271)
 INFO =>     Important Storage Minimum Requirements for Grid & Database Homes
 INFO =>     Most recent ADR incidents for /u01/app/oracle/product/11.2.0/dbhome_1
 INFO =>     Oracle GoldenGate failure prevention best practices
 WARNING =>  Non-default database Services are not configured for orcl
 WARNING =>  Shell limit soft nproc for GI is not configured according to recommendation
 FAIL =>     Database parameter CLUSTER_INTERCONNECTS is not set to the recommended value for orcl
 WARNING =>  Oracle patch 23665623 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 22113854 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 18498878 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 24739928 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 19174639 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 20879889 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 20476175 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 20907061 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 19855835 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 FAIL =>     ASM_DISKSTRING parameter is either null or set to /dev/*
 WARNING =>  ARCHIVELOG mode is disabled for orcl
 FAIL =>     The RMAN snapshot controlfile location is NOT shared on all database nodes in the cluster for orcl
 WARNING =>  OCR is not being backed up daily
 INFO =>     $CRS_HOME/log/hostname/client directory has too many older log files.
 INFO =>     user_dump_dest has trace files older than 30 days for orcl
 INFO =>     background_dump_dest has files older than 30 days for orcl
 INFO =>     At some times checkpoints are not being completed for orcl
 WARNING =>  $ORACLE_HOME/bin/oradism setuid bit is NOT set for /u01/app/oracle/product/11.2.0/dbhome_1
 INFO =>     audit_file_dest  has audit  files older than 30 days for orcl
 WARNING =>  Controlfile is NOT multiplexed for orcl
 WARNING =>  One or more redo log groups are not multiplexed for orcl
 WARNING =>  Oracle database software owner soft nproc shell limit is not configured according to recommendation
 WARNING =>  Database Parameter memory_target is not set to the recommended value on orcl1 instance
 FAIL =>     Operating system hugepages count does not satisfy total SGA requirements
 WARNING =>  NIC bonding is not configured for interconnect
 WARNING =>  NIC bonding is NOT configured for public network (VIP)
 WARNING =>  RAC interconnect network card speed does not meet recommendation
 WARNING =>  OSWatcher is not running as is recommended.
 INFO =>     Jumbo frames (MTU >= 9000) are not configured for interconnect
 WARNING =>  All disk groups should have compatible.rdbms attribute set to recommended values
 WARNING =>  All disk groups should have compatible.advm attribute set to recommended values
 WARNING =>  All disk groups should have compatible.asm attribute set to recommended values
 FAIL =>     Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on orcl1 instance
 FAIL =>     Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on orcl1 instance
 WARNING =>  CSS log file size has not been increased from default (50 MB) .
 WARNING =>  Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for orcl
 FAIL =>     Flashback on PRIMARY is not configured for orcl
 INFO =>     Operational Best Practices
 INFO =>     Database Consolidation Best Practices
 INFO =>     Computer failure prevention best practices
 INFO =>     Data corruption prevention best practices
 INFO =>     Logical corruption prevention best practices
 INFO =>     Database/Cluster/Site failure prevention best practices
 INFO =>     Client failover operational best practices
 WARNING =>  fast_start_mttr_target should be greater than or equal to 300. on orcl1 instance
 INFO =>     Information about hanganalyze and systemstate dump
 WARNING =>  Package -ksh-20120801-10.el6-x86_64 does not meet recommendation for ksh-20100621-12.el6-x86_64
 FAIL =>     Table AUD$[FGA_LOG$] should use Automatic Segment Space Management for orcl
 INFO =>     Database failure prevention best practices
 WARNING =>  Database Archivelog Mode should be set to ARCHIVELOG for orcl
 FAIL =>     Primary database is not protected with Data Guard (standby database) for real-time data protection and availability for orcl
 WARNING =>  Redo log write time is more than 500 milliseconds for orcl
 INFO =>     Parallel Execution Health-Checks and Diagnostics Reports for orcl
 WARNING =>  vm.min_free_kbytes should be set as recommended.
 INFO =>     Oracle recovery manager(rman) best practices
 WARNING =>  RMAN controlfile autobackup should be set to ON for orcl
 INFO =>     Consider increasing the COREDUMPSIZE size
 WARNING =>  Consider investigating changes to the schema objects such as DDLs or new object creation for orcl
 WARNING =>  Consider investigating the frequency of SGA resize operations and take corrective action for orcl
 FAIL =>     One or more network card used by clusterware do not have HOTPLUG=NO
Best Practice checking completed.Checking recommended patches on rac1
--------------------------------------------------------------------------------
Collecting patch inventory on CRS HOME /u01/app/11.2.0/grid_1
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/11.2.0/dbhome_1
--------------------------------------------------------------------------------
1 Recommended CRS patches for 112040 from /u01/app/11.2.0/grid_1 on rac1
--------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description
--------------------------------------------------------------------------------
22502505 no          no    /u01/app/oracle/product/11.2.0/dbhome_1 ACFS PSU 11.2.0.4.160419                                         
26392168 no          no    /u01/app/oracle/product/11.2.0/dbhome_1 DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)                     
26609929 no          no    /u01/app/oracle/product/11.2.0/dbhome_1 OCW PSU 11.2.0.4.170814                                         
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0/dbhome_1 on rac1
--------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description
--------------------------------------------------------------------------------
26392168 no               merge                DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929 no               merge                OCW PSU 11.2.0.4.170814
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
              Clusterware patches summary report
--------------------------------------------------------------------------------
Total patches  Applied on CRS Applied on RDBMS Applied on ASM
--------------------------------------------------------------------------------
1              0              0                0
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
              RDBMS homes patches summary report
--------------------------------------------------------------------------------
Total patches  Applied on RDBMS Applied on ASM ORACLE_HOME
--------------------------------------------------------------------------------
1              2              0                /u01/app/oracle/product/11.2.0/dbhome_1
--------------------------------------------------------------------------------



Copying results from rac2 and generating report. This might take a while. Be patient.


Collecting - ACFS Volumes status
Collecting - Broadcast Requirements for Networks
Collecting - CRS Opatch version
Collecting - CRS user time zone check
Collecting - Disk Information
Collecting - Grid Infastructure user shell limits configuration
Collecting - Interconnect interface config
Collecting - Linux Operating system health check using vmpscan.sh
Collecting - Network interface stats
Collecting - Root user limits
Collecting - Verify no database server kernel out of memory errors
Collecting - root time zone check
Collecting - slabinfo
Collecting - umask setting for GI owner

============================================================
                   Node name - rac2
============================================================

Collecting - /proc/cmdline
Collecting - /proc/modules
Collecting - CPU Information
Collecting - CRS active version
Collecting - CRS oifcfg
Collecting - CRS software version
Collecting - Cluster interconnect (clusterware)
Collecting - Disk I/O Scheduler on Linux
Collecting - DiskFree Information
Collecting - DiskMount Information
Collecting - Huge pages configuration
Collecting - Interconnect network card speed
Collecting - Kernel parameters
Collecting - Linux module config.
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - Maximum number of semaphores per semaphore set
Collecting - Memory Information
Collecting - NUMA Configuration
Collecting - Network Interface Configuration
Collecting - Network Performance
Collecting - Network Service Switch
Collecting - OS Packages
Collecting - OS version
Collecting - Operating system release information and kernel version
Collecting - Oracle Executable Attributes
Collecting - Patches for Grid Infrastructure
Collecting - Patches for RDBMS Home
Collecting - Shared memory segments
Collecting - Table of file system defaults
Collecting - number of semaphore operations per semop system call


Data collections completed. Checking best practices on rac2.
------------------------------------------------------------



 FAIL =>     Bash is vulnerable to code injection (CVE-2014-6271)
 WARNING =>  free physical memory is not at sufficient level on system
 INFO =>     Important Storage Minimum Requirements for Grid & Database Homes
 INFO =>     Most recent ADR incidents for /u01/app/oracle/product/11.2.0/dbhome_1
 INFO =>     Oracle GoldenGate failure prevention best practices
 WARNING =>  Non-default database Services are not configured for orcl
 WARNING =>  Shell limit soft nproc for GI is not configured according to recommendation
 FAIL =>     Database parameter CLUSTER_INTERCONNECTS is not set to the recommended value for orcl
 WARNING =>  Oracle patch 23665623 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 22113854 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 18498878 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 24739928 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 19174639 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 20879889 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 20476175 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 20907061 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 WARNING =>  Oracle patch 19855835 is not applied on RDBMS_HOME /u01/app/oracle/product/11.2.0/dbhome_1
 FAIL =>     ASM_DISKSTRING parameter is either null or set to /dev/*
 FAIL =>     The RMAN snapshot controlfile location is NOT shared on all database nodes in the cluster for orcl
 INFO =>     $CRS_HOME/log/hostname/client directory has too many older log files.
 INFO =>     user_dump_dest has trace files older than 30 days for orcl
 INFO =>     background_dump_dest has files older than 30 days for orcl
 WARNING =>  $ORACLE_HOME/bin/oradism setuid bit is NOT set for /u01/app/oracle/product/11.2.0/dbhome_1
 INFO =>     audit_file_dest  has audit  files older than 30 days for orcl
 WARNING =>  Oracle database software owner soft nproc shell limit is not configured according to recommendation
 WARNING =>  Database Parameter memory_target is not set to the recommended value on orcl2 instance
 FAIL =>     Operating system hugepages count does not satisfy total SGA requirements
 WARNING =>  NIC bonding is not configured for interconnect
 WARNING =>  NIC bonding is NOT configured for public network (VIP)
 WARNING =>  RAC interconnect network card speed does not meet recommendation
 WARNING =>  OSWatcher is not running as is recommended.
 INFO =>     Jumbo frames (MTU >= 9000) are not configured for interconnect
 FAIL =>     Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on orcl2 instance
 FAIL =>     Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on orcl2 instance
 WARNING =>  CSS log file size has not been increased from default (50 MB) .
 WARNING =>  Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for orcl
 WARNING =>  fast_start_mttr_target should be greater than or equal to 300. on orcl2 instance
 WARNING =>  Package -ksh-20120801-10.el6-x86_64 does not meet recommendation for ksh-20100621-12.el6-x86_64
 WARNING =>  Redo log write time is more than 500 milliseconds for orcl
 WARNING =>  vm.min_free_kbytes should be set as recommended.
 INFO =>     Consider increasing the COREDUMPSIZE size
 FAIL =>     One or more network card used by clusterware do not have HOTPLUG=NO
Best Practice checking completed.Checking recommended patches on rac2
--------------------------------------------------------------------------------
Collecting patch inventory on CRS HOME /u01/app/11.2.0/grid_1
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/11.2.0/dbhome_1
--------------------------------------------------------------------------------
1 Recommended CRS patches for 112040 from /u01/app/11.2.0/grid_1 on rac2
--------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description
--------------------------------------------------------------------------------
22502505 no          no    /u01/app/oracle/product/11.2.0/dbhome_1 ACFS PSU 11.2.0.4.160419                                         
26392168 no          no    /u01/app/oracle/product/11.2.0/dbhome_1 DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)                     
26609929 no          no    /u01/app/oracle/product/11.2.0/dbhome_1 OCW PSU 11.2.0.4.170814                                         
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0/dbhome_1 on rac2
--------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description
--------------------------------------------------------------------------------
26392168 no               merge                DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929 no               merge                OCW PSU 11.2.0.4.170814
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
              Clusterware patches summary report
--------------------------------------------------------------------------------
Total patches  Applied on CRS Applied on RDBMS Applied on ASM
--------------------------------------------------------------------------------
1              0              0                0
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
              RDBMS homes patches summary report
--------------------------------------------------------------------------------
Total patches  Applied on RDBMS Applied on ASM ORACLE_HOME
--------------------------------------------------------------------------------
1              2              0                /u01/app/oracle/product/11.2.0/dbhome_1
--------------------------------------------------------------------------------




------------------------------------------------------------
                      CLUSTERWIDE CHECKS
------------------------------------------------------------

------------------------------------------------------------
Detailed report (html) -  /u01/oracle_software/orachk/orachk_rac1_orcl_051618_215243/orachk_rac1_orcl_051618_215243.html


UPLOAD [if required] - /u01/oracle_software/orachk/orachk_rac1_orcl_051618_215243.zip


[root@rac1 orachk]#


=====================================================================

ORAchk Report

======================================================================

Oracle RAC Assessment Report System Health Score is 86 out of 100 (detail)

Cluster Summary

Cluster Name	rac-cluster
OS/Kernel Version	LINUX X86-64 OELRHEL 6 3.8.13-16.2.1.el6uek.x86_64
CRS Home - Version	/u01/app/11.2.0/grid_1 - 11.2.0.4.0
DB Home - Version - Names	/u01/app/oracle/product/11.2.0/dbhome_1 - 11.2.0.4.0 - orcl
Number of nodes	2
Database Servers	2
ORAchk Version	12.2.0.1.4_20171212
Collection	orachk_rac1_orcl_051618_215243
Duration	19 mins, 29 seconds
Executed by	root
Arguments
Collection Date	16-May-2018 21:57:42

Note! This version of ORAchk is considered valid for 0 days from today or until a new version is available

Table of Contents

Database Server

Patch Recommendation

Cluster Wide - All Checks Passed

Maximum Availability Architecture (MAA) Scorecard

Component Elapsed Times

Clusterwide Linux Operating system health check(VMPScan)

Top 10 Time Consuming Checks

Report Feature

 Show Failed checks only

Show checks with the following status:

        Critical  Fail  Warning  Info  Pass

Show details of the following regions:

        Maximum Availability Architecture (MAA) Scorecard
        Patch Recommendation
        Component Elapsed Times

        Clusterwide Linux Operating system health check(VMPScan)

        Top 10 Time Consuming Checks

Show details of the checks:

       Expand AllCollapse All

Show Check Ids

Remove finding from report

Printable View

---

Database Server

Status	Type	Message	Status On	Details
FAIL	OS Check	One or more network card used by clusterware do not have HOTPLUG=NO	All Database Servers	View
FAIL	SQL Check	Table AUD$[FGA_LOG$] should use Automatic Segment Space Management	All Databases	View
FAIL	OS Check	Operating system hugepages count does not satisfy total SGA requirements	All Database Servers	View
FAIL	OS Check	The RMAN snapshot controlfile location is NOT shared on all database nodes in the cluster	All Database Servers	View
FAIL	ASM Check	ASM_DISKSTRING parameter is either null or set to /dev/*	All ASM Instances	View
FAIL	OS Check	Database parameter CLUSTER_INTERCONNECTS is not set to the recommended value	All Database Servers	View
FAIL	OS Check	Bash is vulnerable to code injection (CVE-2014-6271)	All Database Servers	View
WARNING	OS Check	free physical memory is not at sufficient level on system	rac2	View
WARNING	SQL Check	Consider investigating the frequency of SGA resize operations and take corrective action	All Databases	View
WARNING	SQL Check	Consider investigating changes to the schema objects such as DDLs or new object creation	All Databases	View
WARNING	OS Check	vm.min_free_kbytes should be set as recommended.	All Database Servers	View
WARNING	OS Check	Redo log write time is more than 500 milliseconds	All Database Servers	View
WARNING	OS Check	Package -ksh-20120801-10.el6-x86_64 does not meet recommendation for ksh-20100621-12.el6-x86_64	All Database Servers	View
WARNING	OS Check	CSS log file size has not been increased from default (50 MB) .	All Database Servers	View
WARNING	ASM Check	All disk groups should have compatible.asm attribute set to recommended values	All ASM Instances	View
WARNING	ASM Check	All disk groups should have compatible.advm attribute set to recommended values	All ASM Instances	View
WARNING	ASM Check	All disk groups should have compatible.rdbms attribute set to recommended values	All ASM Instances	View
WARNING	OS Check	OSWatcher is not running as is recommended.	All Database Servers	View
WARNING	OS Check	RAC interconnect network card speed does not meet recommendation	All Database Servers	View
WARNING	OS Check	NIC bonding is NOT configured for public network (VIP)	All Database Servers	View
WARNING	OS Check	NIC bonding is not configured for interconnect	All Database Servers	View
WARNING	SQL Parameter Check	Database Parameter memory_target is not set to the recommended value	All Instances	View
WARNING	OS Check	Oracle database software owner soft nproc shell limit is not configured according to recommendation	All Database Servers	View
WARNING	SQL Check	One or more redo log groups are not multiplexed	All Databases	View
WARNING	SQL Check	Controlfile is NOT multiplexed	All Databases	View
WARNING	OS Check	$ORACLE_HOME/bin/oradism setuid bit is NOT set	All Database Servers	View
WARNING	OS Check	OCR is not being backed up daily	All Database Servers	View
WARNING	SQL Check	ARCHIVELOG mode is disabled	All Databases	View
WARNING	Patch Check	Oracle patch 19855835 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 20907061 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 20476175 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 20879889 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 19174639 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 24739928 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 18498878 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 22113854 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 23665623 is not applied on RDBMS_HOME	All Homes	View
WARNING	OS Check	Shell limit soft nproc for GI is not configured according to recommendation	All Database Servers	View
INFO	OS Check	Consider increasing the COREDUMPSIZE size	All Database Servers	View
INFO	OS Check	Parallel Execution Health-Checks and Diagnostics Reports	All Database Servers	View
INFO	OS Check	Information about hanganalyze and systemstate dump	All Database Servers	View
INFO	OS Check	Jumbo frames (MTU >= 9000) are not configured for interconnect	All Database Servers	View
INFO	OS Check	audit_file_dest has audit files older than 30 days	All Database Servers	View
INFO	OS Check	At some times checkpoints are not being completed	rac1	View
INFO	OS Check	background_dump_dest has files older than 30 days	All Database Servers	View
INFO	OS Check	user_dump_dest has trace files older than 30 days	All Database Servers	View
INFO	OS Check	$CRS_HOME/log/hostname/client directory has too many older log files.	All Database Servers	View
INFO	OS Check	Most recent ADR incidents	All Database Servers	View
INFO	OS Check	Important Storage Minimum Requirements for Grid & Database Homes	All Database Servers	View
INFO	ASM Check	Oracle Data Pump Best practices.	All ASM Instances	View
INFO	OS Check	Important Automatic Storage Management (ASM) Notes and Technical White Papers	All Database Servers	View

Cluster Wide

Status	Type	Message	Status On	Details

---

Top

Maximum Availability Architecture (MAA) Scorecard

Outage Type	Status	Type	Message	Status On	Details
DATABASE FAILURE PREVENTION BEST PRACTICES	PASS	[Updated: 03/09/17] Oracle database can be configured with best practices that are applicable to all Oracle databases, including single-instance, Oracle RAC databases, Oracle RAC One Node databases, and the primary and standby databases in Oracle Data Guard or Oracle GoldenGate configurations. Key HA Benefits: (1) Improved recoverability (2) Improved stability
	WARNING SQL Check Database Archivelog Mode should be set to ARCHIVELOG All Databases View
.
COMPUTER FAILURE PREVENTION BEST PRACTICES	INFO	[Updated: 03/09/17] Oracle RAC and Oracle Clusterware allow Oracle Database to run any packaged or custom application across a set of clustered servers. This capability provides server side high availability and scalability. If a clustered server fails, then Oracle Database continues running on the surviving servers. When more processing power is needed, you can add another server without interrupting access to data. Oracle RAC and Oracle Clusterware supports rolling upgrade for most hardware and software changes excluding Oracle RDBMS patch sets and new database releases. To achieve zero or near zero application downtime, integrated client failover best practices need to be incorporated which is covered in client failover operational best practices section of the MAA scorecard. Key HA Benefits: Zero database downtime for node and instance failures. Application brownout can be zero or seconds compared to minutes and an hour with third party cold cluster failover solutions. Zero database and potentially zero application downtime for most planned maintenance activities involving operating system, database or Grid Infrastructure software changes.
WARNING SQL Parameter Check fast_start_mttr_target should be greater than or equal to 300. All Instances View
.
DATA CORRUPTION PREVENTION BEST PRACTICES	PASS	[Updated: 03/09/17] To achieve the most comprehensive data corruption prevention and detection , use Oracle Active Data Guard and configure DB_BLOCK_CHECKSUM,DB_LOST_WRITE_PROTECT and DB_BLOCK_CHECKING database initialization parameters on the primary database and all standby databases in a Data Guard environment. Workload specific testing is required to assess whether the performance overhead with especially DB_BLOCK_CHECKING is acceptable. Using ASM, RMAN, Exadata Storage and Zero Data Loss Recovery Appliance (Recovery Appliance) provide additional data protection checks and repair for Oracle databases and backups. Key HA Benefits Application downtime due to data corruptions can be reduced from hours and days to seconds to no downtime. Prevention, quick detection and fast repair of data block corruptions. With Active Data Guard, physical data block corruptions can be repaired automatically using current blocks from primary or standby databases. With ASM, Exadata Storage and Recovery Appliance, data corruptions can be detected in the background and in some cases repaired automatically. With DB_BLOCK_CHECKING=MED or FULL, logical data block corruptions can be detected and prevented on the primary or standby database.
	FAIL SQL Parameter Check Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value All Instances View FAIL SQL Parameter Check Database parameter DB_BLOCK_CHECKSUM is not set to recommended value All Instances View WARNING OS Check Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. All Database Servers View
.
LOGICAL CORRUPTION PREVENTION BEST PRACTICES	PASS	[Updated: 03/09/17] Oracle Flashback Technology enables fast logical failure repair. Oracle recommends that you use automatic undo management with sufficient space to attain your desired undo retention guarantee, enable Oracle Flashback Database, and allocate sufficient space and I/O bandwidth in the fast recovery area. Application monitoring is required for early detection. Effective and fast repair comes from leveraging and rehearsing the most common application specific logical failures and using the different flashback features effectively (e.g flashback query, flashback version query, flashback transaction query, flashback transaction, flashback drop, flashback table, and flashback database, and 12.2 flashback pluggable database (PDB)). Key HA Benefits: With application monitoring and rehearsed repair actions with flashback technologies, application downtime can reduce from hours and days to the time to detect the logical inconsistency. Fast repair for logical failures caused by malicious or accidental DML or DDL operations. Effect fast point-in-time repair at the appropriate level of granularity: transaction, table, pluggable database, or database. Questions that need to be addressed by your application and operations team: Can your application or monitoring infrastructure detect logical inconsistencies? Is your operations team prepared to use various flashback technologies to repair quickly and efficiently? Is security practices enforced to prevent unauthorized privileges that can result logical inconsistencies?
	FAIL SQL Check Flashback on PRIMARY is not configured All Databases View
.
DATABASE/CLUSTER/SITE FAILURE PREVENTION BEST PRACTICES	INFO	[Updated: 03/09/17] Active Data Guard is the real-time data protection and availability solution that eliminates single point of failure by maintaining one or more synchronized physical replicas of the production database. If an unplanned outage of any kind impacts the production database, applications and users can quickly failover to a synchronized standby, minimizing downtime and preventing data loss. An Active Data Guard standby can be used to offload read-only applications, ad-hoc queries, and backups from the primary database or be dual-purposed as a test system at the same time it provides disaster protection. An Active Data Guard standby can also be used to minimize downtime for planned maintenance when upgrading to new Oracle Database patch sets and major database releases and for select migrations. For zero data loss protection and fastest recovery time, deploy a local Data Guard standby database with Data Guard Fast-Start Failover and integrated client failover. For protection against outages impacting the entire data center, or a broad geography, deploy a Data Guard standby database at a remote location. Oracle 12c Far Sync Standby can achieve zero data loss protection across Wide Area Network or across regions. Key HA Benefits: Active Data Guard provides optimal data protection by using physical replication and comprehensive Oracle validation to maintain an exact byte-for-byte copy of the primary database that can be open read-only to offload reporting, ad-hoc queries and backups. With Oracle 11g release 2 and higher Active Data Guard and real time apply, data block corruptions can be repaired automatically and downtime can be reduced from hours and days of application impact to zero downtime with zero data loss. With MAA best practices, Data Guard Fast-Start Failover and integrated client failover, downtime from database, cluster and site failures can be reduced from hours to days to seconds. With remote standby database (Disaster Recovery Site), you have protection from complete site failures. In all cases, the Active Data Guard instances can be active and used for other activities. Data Guard can reduce risks and downtime for planned maintenance activities by using Database rolling upgrade with transient logical standby or DBMS rolling, standby-first patch apply and database migrations. For other advanced replication requirements where read-write access to a replica database is required while it is being synchronized with the primary database see Oracle GoldenGate logical replication. Oracle GoldenGate can be used to support heterogeneous database platforms and database releases, an effective read-write full or subset logical replica and to reduce or eliminate downtime for application, database or system changes. Oracle GoldenGate flexible logical replication solution's main trade-off is the additional administration for application developer and database administrators.
FAIL SQL Check Primary database is not protected with Data Guard (standby database) for real-time data protection and availability All Databases View
CLIENT FAILOVER OPERATIONAL BEST PRACTICES	PASS	[Updated: 03/09/17] A highly available architecture requires the ability of the application tier to transparently fail over to a surviving instance or database advertising the required service. This ensures that applications are generally available or minimally impacted in the event of node failure, instance failure, or database failures.
	WARNING OS Check Non-default database Services are not configured All Database Servers View
WARNING SQL Check RMAN controlfile autobackup should be set to ON All Databases View
.
ORACLE GOLDENGATE FAILURE PREVENTION BEST PRACTICES	INFO	[Updated: 03/09/17] Oracle GoldenGate is Oracle's strategic logical replication solution for data distribution and data integration. Unlike replication solutions from other vendors, Oracle GoldenGate is more closely integrated with Oracle Database while also providing an open, modular architecture ideal for replication across heterogeneous database management systems. This combination of attributes eliminates compromise, making Oracle GoldenGate the preferred replication solution for addressing requirements that span Oracle Database and non-Oracle Database environments. Key HA Benefits: Potential zero or near zero application downtime for platform migration, database or application upgrades. Active/Active environment with the ability to update both databases. Conflict resolution techniques will need to be considered. To achieve the highest levels of availability resulting in zero or near-zero downtime for both unplanned outages and all planned maintenance activities, customers use the combination of Oracle Active Data Guard and Oracle GoldenGate. Oracle Active Data Guard and Oracle GoldenGate are essential components of the Gold and Platinum MAA tiers.
.
OPERATIONAL BEST PRACTICES	INFO	[Updated: 03/09/17] Operational best practices are an essential prerequisite to high availability.
.
DATABASE CONSOLIDATION BEST PRACTICES	INFO	[Updated: 03/09/17] Database consolidation requires additional planning and management to ensure HA requirements are met.

---

Top

Patch Recommendation

Database server "rac1"

---

1 Recommended CRS patches for 112040 from /u01/app/11.2.0/grid_1

Patch	CRS	ASM	RDBMS	RDBMS_HOME	Patch-Description
22502505	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	ACFS PSU 11.2.0.4.160419
26392168	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	OCW PSU 11.2.0.4.170814

Top 

1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0/dbhome_1

Patch	RDBMS	ASM	Type	Patch-Description
26392168	not-applied	n/a	merge	DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929	not-applied	n/a	merge	OCW PSU 11.2.0.4.170814

Top

Database server "rac2"

---

1 Recommended CRS patches for 112040 from /u01/app/11.2.0/grid_1

Patch	CRS	ASM	RDBMS	RDBMS_HOME	Patch-Description
22502505	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	ACFS PSU 11.2.0.4.160419
26392168	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	OCW PSU 11.2.0.4.170814

Top 

1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0/dbhome_1

Patch	RDBMS	ASM	Type	Patch-Description
26392168	not-applied	n/a	merge	DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929	not-applied	n/a	merge	OCW PSU 11.2.0.4.170814

Top

---

Top

Component Elapsed Times

Component Name	Component Type	Elapsed Time
rac1	Database Server	6 mins, 20 seconds

---

Top

Clusterwide Linux Operating system health check(VMPScan)

Note! This is summary of the VMPScan report. To browse full report, please open ORAchk report present under the 'reports' folder of ORAchk collection, otherwise below links will not work

2 node report generated on: 2018-05-16 22:20:50    Report Name: orachk_rac1_orcl_051618_215243-2018-05-16 22:20:50

HostView (Click hostname for all node parameters)

rac1-2018-05-16-220256	Health (5)	Errors (2)	Warnings (12)
rac2-2018-05-16-220207	Health (5)	Errors (2)	Warnings (11)

ClusterView (Key Parameters)    Clusterview Root

net	os	storage
conf.dns.hostname_ip conf.dns.ns_redundancy conf.dns.resolv_conf conf.gateway.default_gw conf.gateway.default_gwintf conf.gateway.route_-n conf.ntp.active_peers conf.ntp.ntp_redundancy conf.ntp.ntpd_sysv_status conf.ntp.ntpdrift conf.ntp.ntpstat conf.ntp.servers conf.settings.etc_hosts conf.settings.hostname_cmd conf.settings.hosts_localhost conf.settings.ping_localhost dev.conf.brctl_show dev.conf.fullduplex dev.conf.linkactive perf.connectivity.arpinggw perf.connectivity.pinggw perf.netstat.iface_errors	conf.packages.pkg_count conf.sysid.hostname conf.sysid.uname_-r conf.sysvinit.active conf.sysvinit.runlevel hw.cpuinfo.cpuinfo_summary hw.cpuinfo.hyperthreading kernel.conf.etc/sysctl_conf kernel.conf.kernel_sysrq kernel.conf.ulimit_-a logs.system.last_reboots logs.system.log_access logs.system.warnerrors mem.conf.kernel_sem mem.conf.kernel_shmall mem.conf.kernel_shmmax mem.conf.kernel_shmmni mem.numa.numa_active mem.perf.memfree mem.perf.meminfo mem.perf.memtotal perf.process.num_dstates perf.process.uptime perf.process.vmstat_-SMs role.user.id role.vmpscan.precheck role.vmpscan.rootuser time.cron.crond_status time.wallclock.clock time.wallclock.timedatetz	dev.vols.df_-h dev.vols.fdisk_-l dev.vols.lunpath_count dev.vols.mount dev.vols.proc_partitions fs.conf.fstab fs.nfs.exports

---

Top

Top 10 Time Consuming Checks

NOTE: This information is primarily used for helping Oracle optimize the run time of ORAchk.

These timings are not necessarily indicative of any problem and may vary widely from one system to another.

Name	Type	Target	Execution Duration
Patches for Grid Infrastructure	OS Collection	rac2:None	1 mins and 6 secs
Patches for Grid Infrastructure	OS Collection	rac1:None	1 mins and 2 secs
Patches for RDBMS Home	OS Collection	rac2:None	56 secs
OS Packages	OS Collection	rac1:None	45 secs
Patches for RDBMS Home	OS Collection	rac1:None	41 secs
Parallel Execution Health-Checks and Diagnostics Reports	OS Check	rac1:orcl	29 secs
OS Packages	OS Collection	rac2:None	24 secs
Disk I/O Scheduler on Linux	OS Collection	rac1:None	12 secs
Verify Non-Default Database Services Were Created for Each Database	OS Check	rac1:orcl	10 secs
Is ASM storage being used?	OS Check	rac2:orcl	8 secs

Switch to old format

   

I/O Activity Balance Across Disks in ASM

Balanced Oracle ASM Disk Groups

Check cluster name using below command:

[grid@rac1 ~]$ cemutlo -n
rac-cluster
[grid@rac1 ~]$

• Make sure that all the disks in the same Oracle ASM disk group are the same size. Oracle ASM writes in a round-robin fashion to the Oracle ASM disk group disks; therefore, small Oracle ASM disks will be full faster than larger disks, which results in unbalanced I/O activity across the Oracle ASM disks.

• All disks in the same Oracle ASM disk group should have the same performance characteristics

Use below query to find out ASM disks are blanced or not :-

SELECT dg.group_number "GROUP#",
       dg.name,
       DECODE (total_dg.total_io, 0, 100, 100 * (DECODE (SIGN (1 - df.sum_io / total_dg.total_io), -1, 0, (1 - df.sum_io / total_dg.total_io)))) "IO_BALANCED"
  FROM (SELECT d.group_number group_number,
                 SUM (ABS ((d.reads + d.writes) - tot.avg_io)) sum_io
            FROM v$asm_disk_stat d,
                 (SELECT group_number,
                           SUM (reads) + SUM (writes),
                           DECODE (COUNT (*), 0, 0, (SUM (reads) + SUM (writes)) / COUNT (*)) avg_io
                      FROM v$asm_disk_stat
                     WHERE header_status = 'MEMBER'
                  GROUP BY group_number) tot
           WHERE header_status = 'MEMBER' AND tot.group_number = d.group_number
        GROUP BY d.group_number) df,
       (SELECT group_number,
                 SUM (reads) + SUM (writes) total_io
            FROM v$asm_disk_stat
           WHERE header_status = 'MEMBER'
        GROUP BY group_number) total_dg,
        V$ASM_DISKGROUP dg
 WHERE df.group_number = total_dg.group_number
 AND df.group_number = dg.group_number;

In above figure shows an example of my environment where there are two disk groups in the Oracle ASM instance: DATA1 and DATA2. As you can see, the I/O activity is distributed equally across the Oracle ASM disks in each disk group. The conclusion here is that the disk groups (DATA1  and DATA2) are totally balanced.




For more explanation check below link:

https://community.oracle.com/docs/DOC-995178