Step by step troubleshoot performance issue Linux , Oracle - CPU , Memory, I/O

 

🔍 Correlation of ps, top, iostat, and vmstat

Mental Model (Very Important)

Tool	Answers the Question
ps	Which exact process is responsible?
top	Is the problem CPU or memory pressure right now?
iostat	Is storage slow or saturated?
vmstat	Is the kernel under memory / run‑queue / I/O stress?

👉 Never use only one tool.
Real root cause comes from correlating outputs.

---

1️⃣ top — Real‑Time CPU & Memory Pressure

Best usage

top

(or top -o %CPU on newer systems)

What to focus on (top header)

%Cpu(s): 85.2 us, 10.1 sy,  0.0 ni,  2.0 id,  2.5 wa

Field	Meaning
us	User CPU (app / DB code)
sy	Kernel CPU
id	Idle CPU
wa	I/O wait (very important)

Interpretation

• ✅ High us → application or SQL CPU
• ✅ High sy → kernel, system calls, networking
• 🚨 High wa → storage problem, not CPU

---

Process section (top half)

PID   USER  %CPU  %MEM  COMMAND
2456  oracle 180.2 12.3 ora_dbw0

Now you jump to ps.

---

2️⃣ ps — Identify the Exact Culprit

Correlate with:

ps -eo pid,ppid,stat,%cpu,%mem,etime,wchan,comm | sort -k5 -nr | head

Key correlation

top shows	ps confirms
High CPU PID	%cpu, etime
Hung process	STAT = D
Storage wait	wchan = io_schedule

🚨 Example:

2456  D  io_schedule  ora_dbw0

👉 This tells you:
DBWR is blocked on disk I/O

Now you must check storage.

---

3️⃣ iostat — Storage Bottleneck Detection

Best command

iostat -xz 1 5

Critical columns

Column	Meaning
%util	Disk busy time
await	Avg I/O latency (ms)
svctm	Disk service time
r/s w/s	Read/write rate

---

Interpretation Rules (Golden)

Symptom	Meaning
%util > 80%	Disk saturated
await > 20 ms	Storage slow
await >> svctm	Queueing problem
High writes + DBWR stuck	Redo / data disk issue

🚨 Example:

sda  %util=99.8  await=120ms

✅ Confirms ps + top → storage root cause

---

4️⃣ vmstat — Kernel Stress & Memory I/O

Best command

vmstat 1 5

Key columns

r  b   swpd   free   buff  cache  si so   bi bo   in cs us sy id wa

---

Important fields explained

Column	Meaning
r	Run queue (CPU demand)
b	Blocked processes (I/O)
si/so	Swap in/out
bi/bo	Block I/O
wa	I/O wait (kernel view)

---

Correlation logic

vmstat shows	Combined meaning
b > 0	Processes stuck in I/O
wa high	CPU waiting for disk
r > CPU cores	CPU contention
si/so > 0	Memory pressure

🚨 Example:

r=1 b=6 wa=40

👉 Matches:

• ps → many D
• top → high I/O wait
• iostat → high disk latency

🎯 Root cause confirmed: storage

---

5️⃣ End‑to‑End Correlation Scenarios

---

✅ Scenario A: High Load Average

Observations

• uptime → load = 20
• top → CPU idle
• vmstat → b=10, wa=35
• ps → many D state
• iostat → high await

✅ Conclusion
Load is from I/O wait, not CPU
👉 Storage team issue

---

✅ Scenario B: CPU Spike

Observations

• top → %us=90
• vmstat → r > CPU cores
• ps → process in R state
• iostat → normal

✅ Conclusion
Pure CPU problem
👉 Tune SQL / app / threads

---

✅ Scenario C: Hung Oracle Instance

Observations

• ps → ora_dbw0, ora_lgwr in D
• vmstat → b > 5
• iostat → redo disk latency
• top → high wa

✅ Conclusion
Redo or data disk I/O stall
👉 SAN / ASM / NFS issue

---

6️⃣ Golden Troubleshooting Workflow (Memorize This)

symptom →

top →

ps →

vmstat →

iostat →

root cause

One‑liner sequence

top

ps -eo pid,stat,%cpu,wchan,comm | grep D

vmstat 1 5

iostat -xz 1 5

---

✅ Final Cheat Sheet

Tool	Best for
top	Live CPU/memory
ps	Exact process & state
vmstat	Kernel & wait queues
iostat	Disk latency & saturation

🎯 Never trust a single tool
Real diagnosis = correlation

Troubleshooting CPU - I/O : Best linux ps Command Arguments for Troubleshooting - One‑Command “Master View” (Highly Recommended)

 

One‑Command “Master View” (Highly Recommended)

ps -eo user,pid,ppid,stat,%cpu,%mem,etime,lstart,wchan,comm --sort=-%cpu

🔍 What it shows (and why it matters)

Field	Why it’s important
user	Who owns the process
pid	Process ID
ppid	Parent process (helps detect orphans)
stat	Process state (R/S/D/Z/T)
%cpu	CPU consumption
%mem	Memory usage
etime	How long the process has been running
lstart	Exact start time
wchan	Kernel wait channel (I/O diagnosis)
comm	Executable name
--sort=-%cpu	Top CPU consumers first

✅ This is your best single snapshot for general troubleshooting

---

2️⃣ CPU Troubleshooting (High CPU / Run Queues)

ps -eo pid,ppid,stat,psr,pri,ni,%cpu,time,comm --sort=-%cpu | head -20

Key columns

Column	Meaning
psr	Which CPU core it’s running on
pri	Kernel priority
ni	Nice value
time	Total CPU time consumed

✅ Use when:

• Load average is high
• CPU is saturated
• Performance complaints

---

3️⃣ I/O Troubleshooting (MOST CRITICAL)

🔥 Identify blocked processes (D state)

ps -eo pid,stat,wchan,%cpu,etime,comm | awk '$2 ~ /D/'

Why this is powerful

Field	Purpose
D	Uninterruptible sleep (I/O wait)
wchan	What kernel function it’s stuck on
etime	How long it has been blocked

Common wchan values and meaning

wchan	Meaning
io_schedule	Disk I/O wait
wait_on_page_bit	Memory/disk interaction
nfs_wait	NFS hang
blk_mq_get_tag	Storage queue congestion

🚨 If Oracle or DB processes appear here → storage issue almost guaranteed

---

4️⃣ Memory & Leak Detection

ps -eo pid,ppid,stat,rss,vsz,%mem,comm --sort=-rss | head -20

Key fields

Field	Meaning
rss	Real memory in KB
vsz	Virtual memory
%mem	RAM usage

✅ Use when:

• System is swapping
• OOM killer events
• Slow performance despite low CPU

---

5️⃣ Full Command, Arguments & Environment

ps -eo pid,stat,%cpu,%mem,cmd --sort=-%cpu

Why this matters:

• cmd shows complete arguments
• Crucial for:
  • Java tuning
  • Oracle startup flags
  • Application misconfiguration

---

6️⃣ Zombie Process Detection

ps -eo pid,ppid,stat,etime,comm | awk '$3 ~ /Z/'

Why care?

• Zombies indicate parent process bug
• Can exhaust PID space
• Need parent restart (not kill)

---

7️⃣ Oracle / Database‑Focused View (DBA Favorite)

ps -eo pid,stat,%cpu,%mem,etime,wchan,comm | grep ora_

✅ Detects:

• DBWR / LGWR I/O stalls
• Parallel worker hangs
• Backup‑related blockages

---

8️⃣ Thread‑Level Analysis (Advanced CPU Debugging)

ps -eLo pid,lwp,stat,%cpu,psr,comm --sort=-%cpu

Use when:

• Java or Oracle shows high CPU
• Need hot thread detection
• Correlating with perf / jstack

---

9️⃣ Parent‑Child Relationship Analysis

ps -eo pid,ppid,stat,etime,comm --forest

✅ Great for:

• Detecting fork storms
• Tracing hung parent processes
• Understanding service trees

---

10️⃣ Minimal “Health Check” Command (Quick & Safe)

ps -eo pid,stat,%cpu,%mem,etime,comm --sort=-%cpu | head -15

✅ Safe for production
✅ Quick triage
✅ Covers 80% of issues

---

🔑 What to Focus On (Cheat Sheet)

Symptom	Look at
High load	%cpu, R state
Stuck system	D state, wchan
Slowness	%cpu, %mem, etime
Hung DB	ora_* + D
Memory issues	rss, %mem
Defunct processes	Z

---

✅ Final Recommendation (What to Remember)

If you remember only ONE command, make it this:

ps -eo user,pid,ppid,stat,%cpu,%mem,etime,wchan,comm --sort=-%cpu

This single command gives: ✅ CPU
✅ I/O
✅ Memory
✅ State
✅ Ownership
✅ Runtime
✅ Kernel wait reason

Troubleshoot storage I/O performance issue -- Linux , Oracle

Explain end‑to‑end explanation of the command :

    ps -eo pid,stat,comm | grep D

This is a process inspection command used heavily by Linux, Unix, and database administrators for system and performance troubleshooting.

---

1️⃣ What is ps?

ps stands for Process Status.
It reports information about currently running processes on a Linux system.

Think of it as a snapshot of processes at the moment you run the command.

📌 Unlike top or htop, ps:

• Is not interactive
• Shows a point‑in‑time view
• Is ideal for scripting and diagnostics

---

2️⃣ Command Breakdown

ps -eo pid,stat,comm

Let’s split it into parts:

---

🔹 ps

Invokes the process status utility.

---

🔹 -e option (select processes)

-e

Means:
✅ Show all processes running on the system

Without -e, ps would only show processes tied to the current terminal (TTY).

Equivalent options:

ps -e

ps -A

All mean “every process”.

---

🔹 -o option (custom output format)

-o pid,stat,comm

Means:
✅ Choose which columns to display

Instead of default columns, you explicitly request:

Field	Meaning
pid	Process ID
stat	Process state
comm	Command name (executable)

This is extremely useful for focused troubleshooting.

---

3️⃣ Output Columns (Explained in Depth)

🔸 PID — Process ID

Example:

24567

• Unique identifier for a process
• Assigned by the Linux kernel
• Required to manage or inspect processes

Used in commands like:

kill 24567

strace -p 24567

cat /proc/24567/status

📌 Notes:

• PID 1 is always the init/systemd process
• PIDs are reused after processes exit

---

🔸 STAT — Process State (most important field)

The STAT column shows:

1. Main execution state
2. Additional flags

Primary states

Code	Meaning
R	Running or runnable (on CPU or ready)
S	Sleeping (waiting for event)
D	Uninterruptible sleep (I/O wait)
T	Stopped (signal or debugger)
Z	Zombie (dead, not cleaned up)
I	Idle kernel thread (newer kernels)

👉 The first letter is the core state.

---

Modifier flags (can appear after the main letter)

Flag	Meaning
s	Session leader
l	Multithreaded (uses threads)
+	Foreground process
<	High priority
N	Low priority

---

STAT examples explained

Ss

• S → sleeping
• s → session leader
  ✅ Normal background service

Ssl+

• Sleeping
• Session leader
• Multithreaded
• Foreground task
  ✅ Common for DB or Java processes

D

🚨 Critical

• Process waiting on kernel I/O
• Cannot be killed (even kill -9)
• Usually due to:
• Disk I/O
• NFS
• SAN / ASM
• Kernel storage issue

Examples

Ss

→ Sleeping, session leader

D

→ Blocked on I/O (disk, NFS, storage). Very important state

Ssl+

→ Sleeping, session leader, multithreaded, foreground job

📌 Critical note
If a process is in D state, it:

• Cannot be killed (kill -9 won’t work)
• Is usually waiting on disk, SAN, ASM, or NFS
• Indicates storage or kernel-level issues

---

🔸 COMMAND — Executable Name

Example:

oracle
sshd
ora_w00l

• Shows only the binary name
• Does NOT include command‑line arguments

For full command line:

ps -eo pid,stat,cmd

📌 Oracle example:

ora_w00l

Means:

• ora_ → Oracle process
• w00l → Parallel/worker process

---

4️⃣ Sample Output and Interpretation

PID STAT COMMAND

1 Ss systemd

1023 Ssl oracle

2045 D ora_dbw0

How to read this:

• systemd → sleeping session leader (normal)
• oracle → sleeping, multithreaded (normal)
• ora_dbw0 → D state (problem)
  → Indicates disk or ASM issue

---

5️⃣ Why this command is widely used

✅ Lightweight and fast

• No interactive overhead
• Safe on production systems

✅ Perfect for troubleshooting

• Detects:
  • Hung processes
  • Storage stalls
  • Zombie accumulation
  • Oracle background issues

✅ Script‑friendly

Used inside:

• Shell scripts
• Health checks
• Cron jobs

---

6️⃣ Common Enhancements

Show only blocked (D) processes

ps -eo pid,stat,comm | awk '$2 ~ /D/'

---

Sort by process state

ps -eo pid,stat,comm --sort=stat

---

Add user and CPU usage

ps -eo user,pid,stat,%cpu,%mem,comm

---

7️⃣ Practical Use Case (Oracle / DB servers)

DBAs frequently use:

ps -eo pid,stat,comm | grep ora_

To detect:

• Stuck background workers
• DBWR/LGWR waiting on disk
• Parallel query stalls

If many ora_* processes show D: 🚨 Storage team must be involved immediately

---

✅ Final Summary

Component	Purpose
ps	Show process snapshot
-e	Include all processes
-o	Customize output
pid	Process identifier
stat	Execution + wait state
comm	Executable name

🎯 Key troubleshooting signal

• R, S → Normal
• D → I/O or kernel problem
• Z → Parent process issue

• PID → Unique process identifier
• STAT → Current state + extra flags (critical for troubleshooting)
• COMMAND → Executable name

🎯 For troubleshooting:

• R / S → Normal
• D → Investigate immediately
• Z → Parent process issue