ORAchk #2 : Oracle RAC Assessment Report

Oracle RAC Assessment Report System Health Score is 86 out of 100 (detail)

Cluster Summary

Cluster Name	rac-cluster
OS/Kernel Version	LINUX X86-64 OELRHEL 6 3.8.13-16.2.1.el6uek.x86_64
CRS Home - Version	/u01/app/11.2.0/grid_1 - 11.2.0.4.0
DB Home - Version - Names	/u01/app/oracle/product/11.2.0/dbhome_1 - 11.2.0.4.0 - orcl
Number of nodes	2
Database Servers	2
ORAchk Version	12.2.0.1.4_20171212
Collection	orachk_rac1_orcl_051618_215243
Duration	19 mins, 29 seconds
Executed by	root
Arguments
Collection Date	16-May-2018 21:57:42

Note! This version of ORAchk is considered valid for 0 days from today or until a new version is available

Table of Contents

Database Server

Patch Recommendation

Cluster Wide - All Checks Passed

Maximum Availability Architecture (MAA) Scorecard

Component Elapsed Times

Clusterwide Linux Operating system health check(VMPScan)

Top 10 Time Consuming Checks

Report Feature

 Show Failed checks only

Show checks with the following status:

        Critical  Fail  Warning  Info  Pass

Show details of the following regions:

        Maximum Availability Architecture (MAA) Scorecard
        Patch Recommendation
        Component Elapsed Times

        Clusterwide Linux Operating system health check(VMPScan)

        Top 10 Time Consuming Checks

Show details of the checks:

       Expand AllCollapse All

Show Check Ids

Remove finding from report

Printable View

---

Database Server

Status	Type	Message	Status On	Details
FAIL	OS Check	One or more network card used by clusterware do not have HOTPLUG=NO	All Database Servers	View
FAIL	SQL Check	Table AUD$[FGA_LOG$] should use Automatic Segment Space Management	All Databases	View
FAIL	OS Check	Operating system hugepages count does not satisfy total SGA requirements	All Database Servers	View
FAIL	OS Check	The RMAN snapshot controlfile location is NOT shared on all database nodes in the cluster	All Database Servers	View
FAIL	ASM Check	ASM_DISKSTRING parameter is either null or set to /dev/*	All ASM Instances	View
FAIL	OS Check	Database parameter CLUSTER_INTERCONNECTS is not set to the recommended value	All Database Servers	View
FAIL	OS Check	Bash is vulnerable to code injection (CVE-2014-6271)	All Database Servers	View
WARNING	OS Check	free physical memory is not at sufficient level on system	rac2	View
WARNING	SQL Check	Consider investigating the frequency of SGA resize operations and take corrective action	All Databases	View
WARNING	SQL Check	Consider investigating changes to the schema objects such as DDLs or new object creation	All Databases	View
WARNING	OS Check	vm.min_free_kbytes should be set as recommended.	All Database Servers	View
WARNING	OS Check	Redo log write time is more than 500 milliseconds	All Database Servers	View
WARNING	OS Check	Package -ksh-20120801-10.el6-x86_64 does not meet recommendation for ksh-20100621-12.el6-x86_64	All Database Servers	View
WARNING	OS Check	CSS log file size has not been increased from default (50 MB) .	All Database Servers	View
WARNING	ASM Check	All disk groups should have compatible.asm attribute set to recommended values	All ASM Instances	View
WARNING	ASM Check	All disk groups should have compatible.advm attribute set to recommended values	All ASM Instances	View
WARNING	ASM Check	All disk groups should have compatible.rdbms attribute set to recommended values	All ASM Instances	View
WARNING	OS Check	OSWatcher is not running as is recommended.	All Database Servers	View
WARNING	OS Check	RAC interconnect network card speed does not meet recommendation	All Database Servers	View
WARNING	OS Check	NIC bonding is NOT configured for public network (VIP)	All Database Servers	View
WARNING	OS Check	NIC bonding is not configured for interconnect	All Database Servers	View
WARNING	SQL Parameter Check	Database Parameter memory_target is not set to the recommended value	All Instances	View
WARNING	OS Check	Oracle database software owner soft nproc shell limit is not configured according to recommendation	All Database Servers	View
WARNING	SQL Check	One or more redo log groups are not multiplexed	All Databases	View
WARNING	SQL Check	Controlfile is NOT multiplexed	All Databases	View
WARNING	OS Check	$ORACLE_HOME/bin/oradism setuid bit is NOT set	All Database Servers	View
WARNING	OS Check	OCR is not being backed up daily	All Database Servers	View
WARNING	SQL Check	ARCHIVELOG mode is disabled	All Databases	View
WARNING	Patch Check	Oracle patch 19855835 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 20907061 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 20476175 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 20879889 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 19174639 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 24739928 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 18498878 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 22113854 is not applied on RDBMS_HOME	All Homes	View
WARNING	Patch Check	Oracle patch 23665623 is not applied on RDBMS_HOME	All Homes	View
WARNING	OS Check	Shell limit soft nproc for GI is not configured according to recommendation	All Database Servers	View
INFO	OS Check	Consider increasing the COREDUMPSIZE size	All Database Servers	View
INFO	OS Check	Parallel Execution Health-Checks and Diagnostics Reports	All Database Servers	View
INFO	OS Check	Information about hanganalyze and systemstate dump	All Database Servers	View
INFO	OS Check	Jumbo frames (MTU >= 9000) are not configured for interconnect	All Database Servers	View
INFO	OS Check	audit_file_dest has audit files older than 30 days	All Database Servers	View
INFO	OS Check	At some times checkpoints are not being completed	rac1	View
INFO	OS Check	background_dump_dest has files older than 30 days	All Database Servers	View
INFO	OS Check	user_dump_dest has trace files older than 30 days	All Database Servers	View
INFO	OS Check	$CRS_HOME/log/hostname/client directory has too many older log files.	All Database Servers	View
INFO	OS Check	Most recent ADR incidents	All Database Servers	View
INFO	OS Check	Important Storage Minimum Requirements for Grid & Database Homes	All Database Servers	View
INFO	ASM Check	Oracle Data Pump Best practices.	All ASM Instances	View
INFO	OS Check	Important Automatic Storage Management (ASM) Notes and Technical White Papers	All Database Servers	View

Cluster Wide

Status	Type	Message	Status On	Details

---

Top

Maximum Availability Architecture (MAA) Scorecard

Outage Type	Status	Type	Message	Status On	Details
DATABASE FAILURE PREVENTION BEST PRACTICES	PASS	[Updated: 03/09/17] Oracle database can be configured with best practices that are applicable to all Oracle databases, including single-instance, Oracle RAC databases, Oracle RAC One Node databases, and the primary and standby databases in Oracle Data Guard or Oracle GoldenGate configurations. Key HA Benefits: (1) Improved recoverability (2) Improved stability
	WARNING SQL Check Database Archivelog Mode should be set to ARCHIVELOG All Databases View
.
COMPUTER FAILURE PREVENTION BEST PRACTICES	INFO	[Updated: 03/09/17] Oracle RAC and Oracle Clusterware allow Oracle Database to run any packaged or custom application across a set of clustered servers. This capability provides server side high availability and scalability. If a clustered server fails, then Oracle Database continues running on the surviving servers. When more processing power is needed, you can add another server without interrupting access to data. Oracle RAC and Oracle Clusterware supports rolling upgrade for most hardware and software changes excluding Oracle RDBMS patch sets and new database releases. To achieve zero or near zero application downtime, integrated client failover best practices need to be incorporated which is covered in client failover operational best practices section of the MAA scorecard. Key HA Benefits: Zero database downtime for node and instance failures. Application brownout can be zero or seconds compared to minutes and an hour with third party cold cluster failover solutions. Zero database and potentially zero application downtime for most planned maintenance activities involving operating system, database or Grid Infrastructure software changes.
WARNING SQL Parameter Check fast_start_mttr_target should be greater than or equal to 300. All Instances View
.
DATA CORRUPTION PREVENTION BEST PRACTICES	PASS	[Updated: 03/09/17] To achieve the most comprehensive data corruption prevention and detection , use Oracle Active Data Guard and configure DB_BLOCK_CHECKSUM,DB_LOST_WRITE_PROTECT and DB_BLOCK_CHECKING database initialization parameters on the primary database and all standby databases in a Data Guard environment. Workload specific testing is required to assess whether the performance overhead with especially DB_BLOCK_CHECKING is acceptable. Using ASM, RMAN, Exadata Storage and Zero Data Loss Recovery Appliance (Recovery Appliance) provide additional data protection checks and repair for Oracle databases and backups. Key HA Benefits Application downtime due to data corruptions can be reduced from hours and days to seconds to no downtime. Prevention, quick detection and fast repair of data block corruptions. With Active Data Guard, physical data block corruptions can be repaired automatically using current blocks from primary or standby databases. With ASM, Exadata Storage and Recovery Appliance, data corruptions can be detected in the background and in some cases repaired automatically. With DB_BLOCK_CHECKING=MED or FULL, logical data block corruptions can be detected and prevented on the primary or standby database.
	FAIL SQL Parameter Check Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value All Instances View FAIL SQL Parameter Check Database parameter DB_BLOCK_CHECKSUM is not set to recommended value All Instances View WARNING OS Check Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. All Database Servers View
.
LOGICAL CORRUPTION PREVENTION BEST PRACTICES	PASS	[Updated: 03/09/17] Oracle Flashback Technology enables fast logical failure repair. Oracle recommends that you use automatic undo management with sufficient space to attain your desired undo retention guarantee, enable Oracle Flashback Database, and allocate sufficient space and I/O bandwidth in the fast recovery area. Application monitoring is required for early detection. Effective and fast repair comes from leveraging and rehearsing the most common application specific logical failures and using the different flashback features effectively (e.g flashback query, flashback version query, flashback transaction query, flashback transaction, flashback drop, flashback table, and flashback database, and 12.2 flashback pluggable database (PDB)). Key HA Benefits: With application monitoring and rehearsed repair actions with flashback technologies, application downtime can reduce from hours and days to the time to detect the logical inconsistency. Fast repair for logical failures caused by malicious or accidental DML or DDL operations. Effect fast point-in-time repair at the appropriate level of granularity: transaction, table, pluggable database, or database. Questions that need to be addressed by your application and operations team: Can your application or monitoring infrastructure detect logical inconsistencies? Is your operations team prepared to use various flashback technologies to repair quickly and efficiently? Is security practices enforced to prevent unauthorized privileges that can result logical inconsistencies?
	FAIL SQL Check Flashback on PRIMARY is not configured All Databases View
.
DATABASE/CLUSTER/SITE FAILURE PREVENTION BEST PRACTICES	INFO	[Updated: 03/09/17] Active Data Guard is the real-time data protection and availability solution that eliminates single point of failure by maintaining one or more synchronized physical replicas of the production database. If an unplanned outage of any kind impacts the production database, applications and users can quickly failover to a synchronized standby, minimizing downtime and preventing data loss. An Active Data Guard standby can be used to offload read-only applications, ad-hoc queries, and backups from the primary database or be dual-purposed as a test system at the same time it provides disaster protection. An Active Data Guard standby can also be used to minimize downtime for planned maintenance when upgrading to new Oracle Database patch sets and major database releases and for select migrations. For zero data loss protection and fastest recovery time, deploy a local Data Guard standby database with Data Guard Fast-Start Failover and integrated client failover. For protection against outages impacting the entire data center, or a broad geography, deploy a Data Guard standby database at a remote location. Oracle 12c Far Sync Standby can achieve zero data loss protection across Wide Area Network or across regions. Key HA Benefits: Active Data Guard provides optimal data protection by using physical replication and comprehensive Oracle validation to maintain an exact byte-for-byte copy of the primary database that can be open read-only to offload reporting, ad-hoc queries and backups. With Oracle 11g release 2 and higher Active Data Guard and real time apply, data block corruptions can be repaired automatically and downtime can be reduced from hours and days of application impact to zero downtime with zero data loss. With MAA best practices, Data Guard Fast-Start Failover and integrated client failover, downtime from database, cluster and site failures can be reduced from hours to days to seconds. With remote standby database (Disaster Recovery Site), you have protection from complete site failures. In all cases, the Active Data Guard instances can be active and used for other activities. Data Guard can reduce risks and downtime for planned maintenance activities by using Database rolling upgrade with transient logical standby or DBMS rolling, standby-first patch apply and database migrations. For other advanced replication requirements where read-write access to a replica database is required while it is being synchronized with the primary database see Oracle GoldenGate logical replication. Oracle GoldenGate can be used to support heterogeneous database platforms and database releases, an effective read-write full or subset logical replica and to reduce or eliminate downtime for application, database or system changes. Oracle GoldenGate flexible logical replication solution's main trade-off is the additional administration for application developer and database administrators.
FAIL SQL Check Primary database is not protected with Data Guard (standby database) for real-time data protection and availability All Databases View
CLIENT FAILOVER OPERATIONAL BEST PRACTICES	PASS	[Updated: 03/09/17] A highly available architecture requires the ability of the application tier to transparently fail over to a surviving instance or database advertising the required service. This ensures that applications are generally available or minimally impacted in the event of node failure, instance failure, or database failures.
	WARNING OS Check Non-default database Services are not configured All Database Servers View
WARNING SQL Check RMAN controlfile autobackup should be set to ON All Databases View
.
ORACLE GOLDENGATE FAILURE PREVENTION BEST PRACTICES	INFO	[Updated: 03/09/17] Oracle GoldenGate is Oracle's strategic logical replication solution for data distribution and data integration. Unlike replication solutions from other vendors, Oracle GoldenGate is more closely integrated with Oracle Database while also providing an open, modular architecture ideal for replication across heterogeneous database management systems. This combination of attributes eliminates compromise, making Oracle GoldenGate the preferred replication solution for addressing requirements that span Oracle Database and non-Oracle Database environments. Key HA Benefits: Potential zero or near zero application downtime for platform migration, database or application upgrades. Active/Active environment with the ability to update both databases. Conflict resolution techniques will need to be considered. To achieve the highest levels of availability resulting in zero or near-zero downtime for both unplanned outages and all planned maintenance activities, customers use the combination of Oracle Active Data Guard and Oracle GoldenGate. Oracle Active Data Guard and Oracle GoldenGate are essential components of the Gold and Platinum MAA tiers.
.
OPERATIONAL BEST PRACTICES	INFO	[Updated: 03/09/17] Operational best practices are an essential prerequisite to high availability.
.
DATABASE CONSOLIDATION BEST PRACTICES	INFO	[Updated: 03/09/17] Database consolidation requires additional planning and management to ensure HA requirements are met.

---

Top

Patch Recommendation

Database server "rac1"

---

1 Recommended CRS patches for 112040 from /u01/app/11.2.0/grid_1

Patch	CRS	ASM	RDBMS	RDBMS_HOME	Patch-Description
22502505	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	ACFS PSU 11.2.0.4.160419
26392168	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	OCW PSU 11.2.0.4.170814

Top 

1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0/dbhome_1

Patch	RDBMS	ASM	Type	Patch-Description
26392168	not-applied	n/a	merge	DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929	not-applied	n/a	merge	OCW PSU 11.2.0.4.170814

Top

Database server "rac2"

---

1 Recommended CRS patches for 112040 from /u01/app/11.2.0/grid_1

Patch	CRS	ASM	RDBMS	RDBMS_HOME	Patch-Description
22502505	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	ACFS PSU 11.2.0.4.160419
26392168	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929	not-applied	n/a	not-applied	/u01/app/oracle/product/11.2.0/dbhome_1	OCW PSU 11.2.0.4.170814

Top 

1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0/dbhome_1

Patch	RDBMS	ASM	Type	Patch-Description
26392168	not-applied	n/a	merge	DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929	not-applied	n/a	merge	OCW PSU 11.2.0.4.170814

Top

---

Top

Component Elapsed Times

Component Name	Component Type	Elapsed Time
rac1	Database Server	6 mins, 20 seconds

---

Top

Clusterwide Linux Operating system health check(VMPScan)

Note! This is summary of the VMPScan report. To browse full report, please open ORAchk report present under the 'reports' folder of ORAchk collection, otherwise below links will not work

2 node report generated on: 2018-05-16 22:20:50    Report Name: orachk_rac1_orcl_051618_215243-2018-05-16 22:20:50

HostView (Click hostname for all node parameters)

rac1-2018-05-16-220256	Health (5)	Errors (2)	Warnings (12)
rac2-2018-05-16-220207	Health (5)	Errors (2)	Warnings (11)

ClusterView (Key Parameters)    Clusterview Root

net	os	storage
conf.dns.hostname_ip conf.dns.ns_redundancy conf.dns.resolv_conf conf.gateway.default_gw conf.gateway.default_gwintf conf.gateway.route_-n conf.ntp.active_peers conf.ntp.ntp_redundancy conf.ntp.ntpd_sysv_status conf.ntp.ntpdrift conf.ntp.ntpstat conf.ntp.servers conf.settings.etc_hosts conf.settings.hostname_cmd conf.settings.hosts_localhost conf.settings.ping_localhost dev.conf.brctl_show dev.conf.fullduplex dev.conf.linkactive perf.connectivity.arpinggw perf.connectivity.pinggw perf.netstat.iface_errors	conf.packages.pkg_count conf.sysid.hostname conf.sysid.uname_-r conf.sysvinit.active conf.sysvinit.runlevel hw.cpuinfo.cpuinfo_summary hw.cpuinfo.hyperthreading kernel.conf.etc/sysctl_conf kernel.conf.kernel_sysrq kernel.conf.ulimit_-a logs.system.last_reboots logs.system.log_access logs.system.warnerrors mem.conf.kernel_sem mem.conf.kernel_shmall mem.conf.kernel_shmmax mem.conf.kernel_shmmni mem.numa.numa_active mem.perf.memfree mem.perf.meminfo mem.perf.memtotal perf.process.num_dstates perf.process.uptime perf.process.vmstat_-SMs role.user.id role.vmpscan.precheck role.vmpscan.rootuser time.cron.crond_status time.wallclock.clock time.wallclock.timedatetz	dev.vols.df_-h dev.vols.fdisk_-l dev.vols.lunpath_count dev.vols.mount dev.vols.proc_partitions fs.conf.fstab fs.nfs.exports

---

Top

Top 10 Time Consuming Checks

NOTE: This information is primarily used for helping Oracle optimize the run time of ORAchk.

These timings are not necessarily indicative of any problem and may vary widely from one system to another.

Name	Type	Target	Execution Duration
Patches for Grid Infrastructure	OS Collection	rac2:None	1 mins and 6 secs
Patches for Grid Infrastructure	OS Collection	rac1:None	1 mins and 2 secs
Patches for RDBMS Home	OS Collection	rac2:None	56 secs
OS Packages	OS Collection	rac1:None	45 secs
Patches for RDBMS Home	OS Collection	rac1:None	41 secs
Parallel Execution Health-Checks and Diagnostics Reports	OS Check	rac1:orcl	29 secs
OS Packages	OS Collection	rac2:None	24 secs
Disk I/O Scheduler on Linux	OS Collection	rac1:None	12 secs
Verify Non-Default Database Services Were Created for Each Database	OS Check	rac1:orcl	10 secs
Is ASM storage being used?	OS Check	rac2:orcl	8 secs

Switch to new format