Wednesday, May 16, 2018

ORAchk #2 : Oracle RAC Assessment Report



Oracle RAC Assessment Report

System Health Score is 86 out of 100 (detail)

Cluster Summary

Cluster Namerac-cluster
OS/Kernel VersionLINUX X86-64 OELRHEL 6 3.8.13-16.2.1.el6uek.x86_64
CRS Home - Version/u01/app/11.2.0/grid_1 - 11.2.0.4.0
DB Home - Version - Names/u01/app/oracle/product/11.2.0/dbhome_1 - 11.2.0.4.0 - orcl
Number of nodes2
   Database Servers2
ORAchk Version12.2.0.1.4_20171212
Collectionorachk_rac1_orcl_051618_215243
Duration19 mins, 29 seconds
Executed byroot
Arguments
Collection Date16-May-2018 21:57:42

Note! This version of ORAchk is considered valid for 0 days from today or until a new version is available

Table of Contents

  • Database Server
  • Cluster Wide - All Checks Passed

  • Report Feature

  •  

  • Show checks with the following status:
  •                 

  • Show details of the following regions:
  •         
            
            
            
            

  • Show details of the checks:
  •        Expand AllCollapse All


    Database Server

    StatusTypeMessageStatus OnDetails
    FAILOS CheckOne or more network card used by clusterware do not have HOTPLUG=NOAll Database Servers
    FAILSQL CheckTable AUD$[FGA_LOG$] should use Automatic Segment Space ManagementAll Databases
    FAILOS CheckOperating system hugepages count does not satisfy total SGA requirementsAll Database Servers
    FAILOS CheckThe RMAN snapshot controlfile location is NOT shared on all database nodes in the clusterAll Database Servers
    FAILASM CheckASM_DISKSTRING parameter is either null or set to /dev/*All ASM Instances
    FAILOS CheckDatabase parameter CLUSTER_INTERCONNECTS is not set to the recommended valueAll Database Servers
    FAILOS CheckBash is vulnerable to code injection (CVE-2014-6271)All Database Servers
    WARNINGOS Checkfree physical memory is not at sufficient level on systemrac2
    WARNINGSQL CheckConsider investigating the frequency of SGA resize operations and take corrective actionAll Databases
    WARNINGSQL CheckConsider investigating changes to the schema objects such as DDLs or new object creationAll Databases
    WARNINGOS Checkvm.min_free_kbytes should be set as recommended.All Database Servers
    WARNINGOS CheckRedo log write time is more than 500 millisecondsAll Database Servers
    WARNINGOS CheckPackage -ksh-20120801-10.el6-x86_64 does not meet recommendation for ksh-20100621-12.el6-x86_64All Database Servers
    WARNINGOS CheckCSS log file size has not been increased from default (50 MB) .All Database Servers
    WARNINGASM CheckAll disk groups should have compatible.asm attribute set to recommended valuesAll ASM Instances
    WARNINGASM CheckAll disk groups should have compatible.advm attribute set to recommended valuesAll ASM Instances
    WARNINGASM CheckAll disk groups should have compatible.rdbms attribute set to recommended valuesAll ASM Instances
    WARNINGOS CheckOSWatcher is not running as is recommended.All Database Servers
    WARNINGOS CheckRAC interconnect network card speed does not meet recommendationAll Database Servers
    WARNINGOS CheckNIC bonding is NOT configured for public network (VIP)All Database Servers
    WARNINGOS CheckNIC bonding is not configured for interconnectAll Database Servers
    WARNINGSQL Parameter CheckDatabase Parameter memory_target is not set to the recommended valueAll Instances
    WARNINGOS CheckOracle database software owner soft nproc shell limit is not configured according to recommendationAll Database Servers
    WARNINGSQL CheckOne or more redo log groups are not multiplexedAll Databases
    WARNINGSQL CheckControlfile is NOT multiplexedAll Databases
    WARNINGOS Check$ORACLE_HOME/bin/oradism setuid bit is NOT setAll Database Servers
    WARNINGOS CheckOCR is not being backed up dailyAll Database Servers
    WARNINGSQL CheckARCHIVELOG mode is disabledAll Databases
    WARNINGPatch CheckOracle patch 19855835 is not applied on RDBMS_HOMEAll Homes
    WARNINGPatch CheckOracle patch 20907061 is not applied on RDBMS_HOMEAll Homes
    WARNINGPatch CheckOracle patch 20476175 is not applied on RDBMS_HOMEAll Homes
    WARNINGPatch CheckOracle patch 20879889 is not applied on RDBMS_HOMEAll Homes
    WARNINGPatch CheckOracle patch 19174639 is not applied on RDBMS_HOMEAll Homes
    WARNINGPatch CheckOracle patch 24739928 is not applied on RDBMS_HOMEAll Homes
    WARNINGPatch CheckOracle patch 18498878 is not applied on RDBMS_HOMEAll Homes
    WARNINGPatch CheckOracle patch 22113854 is not applied on RDBMS_HOMEAll Homes
    WARNINGPatch CheckOracle patch 23665623 is not applied on RDBMS_HOMEAll Homes
    WARNINGOS CheckShell limit soft nproc for GI is not configured according to recommendationAll Database Servers
    INFOOS CheckConsider increasing the COREDUMPSIZE sizeAll Database Servers
    INFOOS CheckParallel Execution Health-Checks and Diagnostics ReportsAll Database Servers
    INFOOS CheckInformation about hanganalyze and systemstate dumpAll Database Servers
    INFOOS CheckJumbo frames (MTU >= 9000) are not configured for interconnectAll Database Servers
    INFOOS Checkaudit_file_dest has audit files older than 30 daysAll Database Servers
    INFOOS CheckAt some times checkpoints are not being completedrac1
    INFOOS Checkbackground_dump_dest has files older than 30 daysAll Database Servers
    INFOOS Checkuser_dump_dest has trace files older than 30 daysAll Database Servers
    INFOOS Check$CRS_HOME/log/hostname/client directory has too many older log files.All Database Servers
    INFOOS CheckMost recent ADR incidentsAll Database Servers
    INFOOS CheckImportant Storage Minimum Requirements for Grid & Database HomesAll Database Servers
    INFOASM CheckOracle Data Pump Best practices.All ASM Instances
    INFOOS CheckImportant Automatic Storage Management (ASM) Notes and Technical White PapersAll Database Servers

    Cluster Wide

    StatusTypeMessageStatus OnDetails

    Top

    Maximum Availability Architecture (MAA) Scorecard

    Outage TypeStatusTypeMessageStatus OnDetails
    DATABASE FAILURE PREVENTION BEST PRACTICESPASS
     [Updated: 03/09/17]
    
    Oracle database can be configured with best practices that are applicable to all Oracle databases, including single-instance, Oracle RAC databases, Oracle RAC One Node databases, and the primary and standby databases in Oracle Data Guard or Oracle GoldenGate configurations.
    
    Key HA Benefits:
    
    (1) Improved recoverability
    (2) Improved stability
    
     
    
    WARNINGSQL CheckDatabase Archivelog Mode should be set to ARCHIVELOGAll Databases
    .
    COMPUTER FAILURE PREVENTION BEST PRACTICESINFO
     [Updated: 03/09/17]
    
    Oracle RAC and Oracle Clusterware allow Oracle Database to run any packaged or custom application across a set of clustered servers. This capability provides server side high availability and scalability. If a clustered server fails, then Oracle Database continues running on the surviving servers. When more processing power is needed, you can add another server without interrupting access to data.
    
    
    Oracle RAC and Oracle Clusterware supports rolling upgrade for most hardware and software changes excluding Oracle RDBMS patch sets and new database releases.
    
    
     To achieve zero or near zero application downtime, integrated client failover best practices need to be incorporated which is covered in client failover operational best practices section of the MAA scorecard.
    
    
    Key HA Benefits:
    
    • Zero database downtime for node and instance failures. Application brownout can be zero or seconds compared to minutes and an hour with third party cold cluster failover solutions.
    • Zero database and potentially zero application downtime for most planned maintenance activities involving operating system, database or Grid Infrastructure software changes.
    WARNINGSQL Parameter Checkfast_start_mttr_target should be greater than or equal to 300.All Instances
    .
    DATA CORRUPTION PREVENTION BEST PRACTICESPASS
     [Updated: 03/09/17]
    
    To achieve the most comprehensive data corruption prevention and detection , use Oracle Active Data Guard and configure DB_BLOCK_CHECKSUM,DB_LOST_WRITE_PROTECT and DB_BLOCK_CHECKING database initialization parameters on the primary database and all standby databases in a Data Guard environment. Workload specific testing is required to assess whether the performance overhead with especially DB_BLOCK_CHECKING is acceptable.
    
    Using ASM, RMAN, Exadata Storage and Zero Data Loss Recovery Appliance (Recovery Appliance) provide additional data protection checks and repair for Oracle databases and backups.    
    
     Key HA Benefits
    
    • Application downtime due to data corruptions can be reduced from hours and days to seconds to no downtime.
    • Prevention, quick detection and fast repair of data block corruptions.
    • With Active Data Guard, physical data block corruptions can be repaired automatically using current blocks from primary or standby databases.
    • With ASM, Exadata Storage and Recovery Appliance, data corruptions can be detected in the background and in some cases repaired automatically.
    • With DB_BLOCK_CHECKING=MED or FULL, logical data block corruptions can be detected and prevented on the primary or standby database.
    FAILSQL Parameter CheckDatabase parameter DB_LOST_WRITE_PROTECT is not set to recommended valueAll Instances
    FAILSQL Parameter CheckDatabase parameter DB_BLOCK_CHECKSUM is not set to recommended valueAll Instances
    WARNINGOS CheckDatabase parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value.All Database Servers
    .
    LOGICAL CORRUPTION PREVENTION BEST PRACTICESPASS
     [Updated: 03/09/17]
    
    Oracle Flashback Technology enables fast logical failure repair. Oracle recommends that you use automatic undo management with sufficient space to attain your desired undo retention guarantee, enable Oracle Flashback Database, and allocate sufficient space and I/O bandwidth in the fast recovery area.  Application monitoring is required for early detection.  Effective and fast repair comes from leveraging and rehearsing the most common application specific logical failures and using the different flashback features effectively (e.g flashback query, flashback version query, flashback transaction query, flashback transaction, flashback drop, flashback table, and flashback database, and 12.2 flashback pluggable database (PDB)).
    
    Key HA Benefits:
    
    • With application monitoring and rehearsed repair actions with flashback technologies, application downtime can reduce from hours and days to the time to detect the logical inconsistency.
    • Fast repair for logical failures caused by malicious or accidental DML or DDL operations.
    • Effect fast point-in-time repair at the appropriate level of granularity: transaction, table, pluggable database, or database.
    Questions that need to be addressed by your application and operations team:
    1. Can your application or monitoring infrastructure detect logical inconsistencies?
    2. Is your operations team prepared to use various flashback technologies to repair quickly and efficiently?
    3. Is security practices enforced to prevent unauthorized privileges that can result logical inconsistencies?
    FAILSQL CheckFlashback on PRIMARY is not configuredAll Databases
    .
    DATABASE/CLUSTER/SITE FAILURE PREVENTION BEST PRACTICESINFO
     [Updated: 03/09/17]
    
    Active Data Guard is the real-time data protection and availability solution that eliminates single point of failure by maintaining one or more synchronized physical replicas of the production database. If an unplanned outage of any kind impacts the production database, applications and users can quickly failover to a synchronized standby, minimizing downtime and preventing data loss. An Active Data Guard standby can be used to offload read-only applications, ad-hoc queries, and backups from the primary database or be dual-purposed as a test system at the same time it provides disaster protection. An Active Data Guard standby can also be used to minimize downtime for planned maintenance when upgrading to new Oracle Database patch sets and major database releases and for select migrations.  
     
    For zero data loss protection and fastest recovery time, deploy a local Data Guard standby database with Data Guard Fast-Start Failover and integrated client failover. For protection against outages impacting the entire data center, or a broad geography, deploy a Data Guard standby database at a remote location. Oracle 12c Far Sync Standby can achieve zero data loss protection across Wide Area Network or across regions.
    
    Key HA Benefits:
    
    • Active Data Guard provides optimal data protection by using physical replication and comprehensive Oracle validation to maintain an exact byte-for-byte copy of the primary database that can be open read-only to offload reporting, ad-hoc queries and backups.
    • With Oracle 11g release 2 and higher Active Data Guard and real time apply, data block corruptions can be repaired automatically and downtime can be reduced from hours and days of application impact to zero downtime with zero data loss.
    • With MAA best practices, Data Guard Fast-Start Failover and integrated client failover, downtime from database, cluster and site failures can be reduced from hours to days to seconds.
    • With remote standby database (Disaster Recovery Site), you have protection from complete site failures.
    • In all cases, the Active Data Guard instances can be active and used for other activities.
    • Data Guard can reduce risks and downtime for planned maintenance activities by using Database rolling upgrade with transient logical standby or DBMS rolling, standby-first patch apply and database migrations.
    • For other advanced replication requirements where read-write access to a replica database is required while it is being synchronized with the primary database see Oracle GoldenGate logical replication. Oracle GoldenGate can be used to support heterogeneous database platforms and database releases, an effective read-write full or subset logical replica and to reduce or eliminate downtime for application, database or system changes. Oracle GoldenGate flexible logical replication solution's main trade-off is the additional administration for application developer and database administrators.
    FAILSQL CheckPrimary database is not protected with Data Guard (standby database) for real-time data protection and availabilityAll Databases
    CLIENT FAILOVER OPERATIONAL BEST PRACTICESPASS
     [Updated: 03/09/17]
    
    A highly available architecture requires the ability of the application tier to transparently fail over to a surviving instance or database advertising the required service. This ensures that applications are generally available or minimally impacted in the event of node failure, instance failure, or database failures.
     
    
    WARNINGOS CheckNon-default database Services are not configuredAll Database Servers
    WARNINGSQL CheckRMAN controlfile autobackup should be set to ONAll Databases
    .
    ORACLE GOLDENGATE FAILURE PREVENTION BEST PRACTICESINFO
     [Updated: 03/09/17]
    
    Oracle GoldenGate is Oracle's strategic logical replication solution for data distribution and data integration. Unlike replication solutions from other vendors, Oracle GoldenGate is more closely integrated with Oracle Database while also providing an open, modular architecture ideal for replication across heterogeneous database management systems. This combination of attributes eliminates compromise, making Oracle GoldenGate the preferred replication solution for addressing requirements that span Oracle Database and non-Oracle Database environments.
    
    Key HA Benefits:
    
    • Potential zero or near zero application downtime for platform migration, database or application upgrades.
    • Active/Active environment with the ability to update both databases. Conflict resolution techniques will need to be considered.
    To achieve the highest levels of availability resulting in zero or near-zero downtime for both unplanned outages and all planned maintenance activities, customers use the combination of Oracle Active Data Guard and Oracle GoldenGate. Oracle Active Data Guard and Oracle GoldenGate are essential components of the Gold and Platinum MAA tiers.
    .
    OPERATIONAL BEST PRACTICESINFO
     [Updated: 03/09/17]
    
    Operational best practices are an essential prerequisite to high availability.
     
    
    .
    DATABASE CONSOLIDATION BEST PRACTICESINFO
     [Updated: 03/09/17]
    
    Database consolidation requires additional planning and management to ensure HA requirements are met.
     
    

    Top

    Patch Recommendation

    Database server "rac1"




    1 Recommended CRS patches for 112040 from /u01/app/11.2.0/grid_1
    PatchCRSASMRDBMSRDBMS_HOMEPatch-Description
    22502505not-appliedn/anot-applied/u01/app/oracle/product/11.2.0/dbhome_1ACFS PSU 11.2.0.4.160419
    26392168not-appliedn/anot-applied/u01/app/oracle/product/11.2.0/dbhome_1DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
    26609929not-appliedn/anot-applied/u01/app/oracle/product/11.2.0/dbhome_1OCW PSU 11.2.0.4.170814
    Top 

    1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0/dbhome_1
    PatchRDBMSASMTypePatch-Description
    26392168not-appliedn/amergeDB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
    26609929not-appliedn/amergeOCW PSU 11.2.0.4.170814
    Top

    Database server "rac2"




    1 Recommended CRS patches for 112040 from /u01/app/11.2.0/grid_1
    PatchCRSASMRDBMSRDBMS_HOMEPatch-Description
    22502505not-appliedn/anot-applied/u01/app/oracle/product/11.2.0/dbhome_1ACFS PSU 11.2.0.4.160419
    26392168not-appliedn/anot-applied/u01/app/oracle/product/11.2.0/dbhome_1DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
    26609929not-appliedn/anot-applied/u01/app/oracle/product/11.2.0/dbhome_1OCW PSU 11.2.0.4.170814
    Top 

    1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0/dbhome_1
    PatchRDBMSASMTypePatch-Description
    26392168not-appliedn/amergeDB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
    26609929not-appliedn/amergeOCW PSU 11.2.0.4.170814
    Top

    Top

    Component Elapsed Times

    Component NameComponent TypeElapsed Time
    rac1Database Server6 mins, 20 seconds

    Top

    Clusterwide Linux Operating system health check(VMPScan)

    Note! This is summary of the VMPScan report. To browse full report, please open ORAchk report present under the 'reports' folder of ORAchk collection, otherwise below links will not work


    2 node report generated on: 2018-05-16 22:20:50    Report Name: orachk_rac1_orcl_051618_215243-2018-05-16 22:20:50



    Top

    Top 10 Time Consuming Checks

    NOTE: This information is primarily used for helping Oracle optimize the run time of ORAchk.

    These timings are not necessarily indicative of any problem and may vary widely from one system to another.
    NameTypeTargetExecution Duration
    Patches for Grid InfrastructureOS Collectionrac2:None1 mins and 6 secs
    Patches for Grid InfrastructureOS Collectionrac1:None1 mins and 2 secs
    Patches for RDBMS HomeOS Collectionrac2:None56 secs
    OS PackagesOS Collectionrac1:None45 secs
    Patches for RDBMS HomeOS Collectionrac1:None41 secs
    Parallel Execution Health-Checks and Diagnostics ReportsOS Checkrac1:orcl29 secs
    OS PackagesOS Collectionrac2:None24 secs
    Disk I/O Scheduler on LinuxOS Collectionrac1:None12 secs
    Verify Non-Default Database Services Were Created for Each DatabaseOS Checkrac1:orcl10 secs
    Is ASM storage being used?OS Checkrac2:orcl8 secs

    Switch to new format