Step by step troubleshoot performance issue Linux , Oracle - CPU , Memory, I/O

 

🔍 Correlation of ps, top, iostat, and vmstat

Mental Model (Very Important)

Tool	Answers the Question
ps	Which exact process is responsible?
top	Is the problem CPU or memory pressure right now?
iostat	Is storage slow or saturated?
vmstat	Is the kernel under memory / run‑queue / I/O stress?

👉 Never use only one tool.
Real root cause comes from correlating outputs.

---

1️⃣ top — Real‑Time CPU & Memory Pressure

Best usage

top

(or top -o %CPU on newer systems)

What to focus on (top header)

%Cpu(s): 85.2 us, 10.1 sy,  0.0 ni,  2.0 id,  2.5 wa

Field	Meaning
us	User CPU (app / DB code)
sy	Kernel CPU
id	Idle CPU
wa	I/O wait (very important)

Interpretation

• ✅ High us → application or SQL CPU
• ✅ High sy → kernel, system calls, networking
• 🚨 High wa → storage problem, not CPU

---

Process section (top half)

PID   USER  %CPU  %MEM  COMMAND
2456  oracle 180.2 12.3 ora_dbw0

Now you jump to ps.

---

2️⃣ ps — Identify the Exact Culprit

Correlate with:

ps -eo pid,ppid,stat,%cpu,%mem,etime,wchan,comm | sort -k5 -nr | head

Key correlation

top shows	ps confirms
High CPU PID	%cpu, etime
Hung process	STAT = D
Storage wait	wchan = io_schedule

🚨 Example:

2456  D  io_schedule  ora_dbw0

👉 This tells you:
DBWR is blocked on disk I/O

Now you must check storage.

---

3️⃣ iostat — Storage Bottleneck Detection

Best command

iostat -xz 1 5

Critical columns

Column	Meaning
%util	Disk busy time
await	Avg I/O latency (ms)
svctm	Disk service time
r/s w/s	Read/write rate

---

Interpretation Rules (Golden)

Symptom	Meaning
%util > 80%	Disk saturated
await > 20 ms	Storage slow
await >> svctm	Queueing problem
High writes + DBWR stuck	Redo / data disk issue

🚨 Example:

sda  %util=99.8  await=120ms

✅ Confirms ps + top → storage root cause

---

4️⃣ vmstat — Kernel Stress & Memory I/O

Best command

vmstat 1 5

Key columns

r  b   swpd   free   buff  cache  si so   bi bo   in cs us sy id wa

---

Important fields explained

Column	Meaning
r	Run queue (CPU demand)
b	Blocked processes (I/O)
si/so	Swap in/out
bi/bo	Block I/O
wa	I/O wait (kernel view)

---

Correlation logic

vmstat shows	Combined meaning
b > 0	Processes stuck in I/O
wa high	CPU waiting for disk
r > CPU cores	CPU contention
si/so > 0	Memory pressure

🚨 Example:

r=1 b=6 wa=40

👉 Matches:

• ps → many D
• top → high I/O wait
• iostat → high disk latency

🎯 Root cause confirmed: storage

---

5️⃣ End‑to‑End Correlation Scenarios

---

✅ Scenario A: High Load Average

Observations

• uptime → load = 20
• top → CPU idle
• vmstat → b=10, wa=35
• ps → many D state
• iostat → high await

✅ Conclusion
Load is from I/O wait, not CPU
👉 Storage team issue

---

✅ Scenario B: CPU Spike

Observations

• top → %us=90
• vmstat → r > CPU cores
• ps → process in R state
• iostat → normal

✅ Conclusion
Pure CPU problem
👉 Tune SQL / app / threads

---

✅ Scenario C: Hung Oracle Instance

Observations

• ps → ora_dbw0, ora_lgwr in D
• vmstat → b > 5
• iostat → redo disk latency
• top → high wa

✅ Conclusion
Redo or data disk I/O stall
👉 SAN / ASM / NFS issue

---

6️⃣ Golden Troubleshooting Workflow (Memorize This)

symptom →

top →

ps →

vmstat →

iostat →

root cause

One‑liner sequence

top

ps -eo pid,stat,%cpu,wchan,comm | grep D

vmstat 1 5

iostat -xz 1 5

---

✅ Final Cheat Sheet

Tool	Best for
top	Live CPU/memory
ps	Exact process & state
vmstat	Kernel & wait queues
iostat	Disk latency & saturation

🎯 Never trust a single tool
Real diagnosis = correlation