Showing posts with label PostgreSQL. Show all posts
Showing posts with label PostgreSQL. Show all posts

Saturday, November 14, 2026

PostgreSQL Process Architecture Diagram and Details


PostgreSQL Process Architecture Diagram

                           +------------------+
                           |    POSTMASTER    |
                           |   (Parent PID)   |
                           +---------+--------+
                                     |
       ----------------------------------------------------------------
       |            |            |            |            |            |
       v            v            v            v            v            v
+-------------+ +-----------+ +-----------+ +-----------+ +---------+ +---------+
| Background  | | WAL       | |Checkpointer| | Autovac  | | Archiver| | Startup |
| Writer      | | Writer    | |           | | Launcher | |         | | Process |
+-------------+ +-----------+ +-----------+ +-----------+ +---------+ +---------+
                                                        |
                                            ----------------------
                                            |         |          |
                                            v         v          v
                                      +----------+ +----------+ +----------+
                                      |Vacuum W1 | |Vacuum W2 | |Vacuum W3 |
                                      +----------+ +----------+ +----------+
                                        Client Connections
                                                |
       ----------------------------------------------------------------
       |                 |                    |                  |
       v                 v                    v                  v
+--------------+ +--------------+ +--------------+ +--------------+
| Backend PID  | | Backend PID  | | Backend PID  | | Backend PID  |
| User Session | | User Session | | User Session | | User Session |
+--------------+ +--------------+ +--------------+ +--------------+
Shared Memory:
--------------------------------------------------------
| Shared Buffers | WAL Buffers | Locks | Proc Array |
--------------------------------------------------------


How to See PostgreSQL Processes at OS Level

Find Postmaster (Parent Process)

ps -ef | grep postgres

Example:

postgres  1000     1  0 10:00 ? 00:00:00 postgres
postgres  1001  1000  0 10:00 ? 00:00:00 postgres: checkpointer
postgres  1002  1000  0 10:00 ? 00:00:00 postgres: background writer
postgres  1003  1000  0 10:00 ? 00:00:00 postgres: walwriter
postgres  1004  1000  0 10:00 ? 00:00:00 postgres: autovacuum launcher

Notice:

PID 1000 = Parent Process
PPID 1   = Started by OS

This is the Postmaster.


Better Process Tree

Linux

pstree -p | grep postgres

Example:

postgres(1000)
 ├─postgres(1001)
 ├─postgres(1002)
 ├─postgres(1003)
 ├─postgres(1004)
 ├─postgres(1200)
 ├─postgres(1201)
 └─postgres(1202)

Much easier to visualize.


1. Postmaster

Responsibility

  • Starts all PostgreSQL processes
  • Accepts client connections
  • Handles crash recovery
  • Monitors child processes

Check:

ps -fp <postmaster_pid>

Example:

ps -fp 1000


If Backend Crashes

Backend dies
      ↓
Postmaster detects
      ↓
May restart required processes

Postmaster acts like a supervisor process.


2. Backend Process

Every client connection gets a backend process.

Connect:

psql postgres

Run:

SELECT pg_backend_pid();

Example:

  pg_backend_pid
-----------------
      14567

At OS level:

ps -fp 14567

Output:

postgres: appuser mydb 10.0.0.10 idle


Find Active Queries

SELECT pid,
       usename,
       state,
       query
FROM pg_stat_activity;

Map PID to Linux:

top -Hp <pid>

or

ps -fp <pid>


3. WAL Writer

Responsibility

Writes WAL buffers to:

$PGDATA/pg_wal

Process appears as:

postgres: walwriter

Check:

ps -ef | grep walwriter

Example:

postgres 1003 1000 0 ? postgres: walwriter


Observe WAL Activity

Inside PostgreSQL:

SELECT *
FROM pg_stat_wal;

Important metrics:

wal_records
wal_fpi
wal_bytes


4. Checkpointer

Responsibility

Writes dirty pages from memory to data files.

Process:

postgres: checkpointer

View:

ps -ef | grep checkpointer


Monitor

SELECT *
FROM pg_stat_bgwriter;

Interesting columns:

checkpoints_timed
checkpoints_req
checkpoint_write_time
checkpoint_sync_time


5. Background Writer

Responsibility

Gradually writes dirty buffers before checkpoint.

Goal:

Reduce checkpoint spikes

Process:

postgres: background writer

Check:

ps -ef | grep "background writer"


Why Important

Without BGWriter:

Checkpoint writes millions of dirty pages

causing:

High I/O
Slow performance


6. Autovacuum Launcher

Responsibility

Monitors all databases.

Starts worker processes when necessary.

OS command:

ps -ef | grep autovacuum

Example:

postgres: autovacuum launcher


7. Autovacuum Workers

Process:

postgres: autovacuum worker

Check:

ps -ef | grep "autovacuum worker"

Example:

postgres: autovacuum worker mydb


Monitor Current Vacuum

SELECT *
FROM pg_stat_progress_vacuum;

Output:

pid
phase
heap_blks_total
heap_blks_scanned


8. Archiver Process

Only if:

archive_mode = on

Process:

postgres: archiver

Check:

ps -ef | grep archiver


Monitoring

SELECT *
FROM pg_stat_archiver;

Example:

archived_count
failed_count


9. Startup Process

Visible during:

Database startup
Crash recovery
Standby recovery

OS:

ps -ef | grep startup

Example:

postgres: startup recovering


During Recovery

Logs:

tail -f $PGDATA/log/postgresql.log

Example:

redo starts at ...
redo done at ...


10. WAL Sender

Primary Server Replication.

Process:

postgres: walsender

Check:

ps -ef | grep walsender


Monitoring

SELECT *
FROM pg_stat_replication;

Example:

application_name
client_addr
state
sync_state


11. WAL Receiver

Standby Server.

Process:

postgres: walreceiver

OS:

ps -ef | grep walreceiver


Monitoring

SELECT *
FROM pg_stat_wal_receiver;


Real Production Troubleshooting

Check Entire Process Architecture

ps -ef | grep postgres


Check Process Tree

pstree -p | grep postgres


Check CPU Consumer

top -p $(pgrep postgres | tr '\n' ',')


Match PostgreSQL PID with Query

SELECT pid,
       usename,
       state,
       wait_event,
       query
FROM pg_stat_activity;

Then:

ps -fp <pid>


Check Shared Memory

ipcs -m

Or:

cat /proc/<postmaster_pid>/status


Oracle vs PostgreSQL Process Mapping

PostgreSQLOracle
PostmasterPMON (roughly supervisor role)
Backend ProcessDedicated Server Process
WAL WriterLGWR
CheckpointerCKPT
Background WriterDBWR
Startup RecoverySMON
WAL SenderData Guard Transport
WAL ReceiverManaged Recovery Process
ArchiverARCn
AutovacuumNo direct equivalent

Interview Answer (2-Minute Version)

PostgreSQL uses a process-based architecture. The Postmaster is the parent process that starts and monitors all child processes. For each client connection, a dedicated backend process is created. Key background processes include WAL Writer (writes WAL records for durability), Checkpointer (flushes dirty pages to disk), Background Writer (reduces checkpoint I/O spikes), Autovacuum Launcher and Workers (clean dead tuples and freeze old rows), Archiver (archives WAL files), and WAL Sender/Receiver (streaming replication). At the OS level these can be viewed using commands such as:

ps -ef | grep postgres
pstree -p | grep postgres
top -H -p <pid>

and correlated with database views like:

pg_stat_activity
pg_stat_progress_vacuum
pg_stat_replication
pg_stat_bgwriter






About the Author : Anurag Pandey | Database cloud Architect | Oracle | SQL Server | PostgreSQL | AI and Data Platform Enthusiast

Anurag Pandey is a Database Architect with expertise in Oracle, PostgreSQL, SQL Server, Linux, and enterprise database platforms and specializes in database architecture, performance tuning, high availability, disaster recovery, cloud modernization, and operational excellence. With extensive experience in enterprise-scale environments, Anurag is passionate about leveraging AI, automation, and modern data platforms to improve reliability, scalability, and business outcomes. He actively contributes to technical communities through knowledge sharing, architecture guidance, and thought leadership in database and data engineering domains.

Follow more articles at

Visit Blog

PostgreSQL Postmaster and Background Processes

 

PostgreSQL Postmaster and Background Processes

PostgreSQL has a multi-process architecture. Unlike Oracle (which is largely multi-threaded), PostgreSQL creates multiple OS processes.

1. Postmaster (Main PostgreSQL Process)

The Postmaster is the parent process of all PostgreSQL processes.

When PostgreSQL starts:

pg_ctl start

or

systemctl start postgresql

the first process created is:

Postmaster

Its responsibilities:

  • Listen for client connections
  • Spawn backend processes
  • Start background processes
  • Monitor child processes
  • Perform crash recovery
  • Restart failed workers

Architecture:

                    Postmaster
                         |
    -------------------------------------------------
    |          |           |          |             |
 Background  WAL       Checkpoint   Autovacuum   Backend
 Writer      Writer     Process      Workers     Sessions


2. Backend Processes

Every client connection gets its own backend process.

Example:

Application User-1
        |
    Backend PID 1001
Application User-2
        |
    Backend PID 1002
Application User-3
        |
    Backend PID 1003

Query:

SELECT pid,
       usename,
       application_name
FROM pg_stat_activity;

Each row corresponds to a backend process.


Important Background Processes

3. Background Writer (BgWriter)

Purpose:

Reduce checkpoint spikes.

When users modify data:

Shared Buffers
      ↓
Dirty Pages

These pages stay in memory.

The Background Writer gradually writes dirty pages to disk.

Without it:

Checkpoint would write everything

causing sudden I/O spikes.

View:

SELECT *
FROM pg_stat_bgwriter;


4. WAL Writer

One of the most important processes.

Purpose:

Flush WAL buffers to WAL files.

Flow:

INSERT/UPDATE/DELETE
       ↓
WAL Buffer
       ↓
WAL Writer
       ↓
pg_wal/

Example:

Shared Memory
      |
 WAL Buffer
      |
 WAL Writer
      |
 pg_wal files

Without WAL Writer:

Crash recovery impossible


5. Checkpointer

Purpose:

Create checkpoints

Checkpoint means:

Write dirty pages
to disk permanently

Flow:

Shared Buffers
      ↓
Disk Files

Checkpoint record also goes into WAL.

Benefits:

Faster crash recovery

Check settings:

SHOW checkpoint_timeout;
SHOW max_wal_size;

Monitor:

SELECT *
FROM pg_stat_bgwriter;


6. Autovacuum Launcher

Purpose:

Find tables needing vacuum.

It doesn't vacuum itself.

Rather:

Launcher
     ↓
Creates Workers

Architecture:

Autovacuum Launcher
        |
        +---- Worker 1
        +---- Worker 2
        +---- Worker 3

Check:

SHOW autovacuum;


7. Autovacuum Workers

Responsibilities:

  • Remove dead tuples
  • Update statistics
  • Freeze old tuples
  • Prevent XID wraparound

Example:

SELECT *
FROM pg_stat_progress_vacuum;

Shows currently running workers.


8. Statistics Collector (Older Versions)

In PostgreSQL before v14:

Stats Collector Process

maintained statistics used by:

pg_stat_activity
pg_stat_database
pg_stat_user_tables

In modern PostgreSQL versions:

Shared Memory Statistics

replaces much of the old collector architecture.


9. Archiver Process

Used when:

archive_mode = on

Purpose:

Copy WAL files
to archive storage

Used for:

PITR
Disaster Recovery

Check:

SELECT *
FROM pg_stat_archiver;

Example:

pg_wal
   ↓
Archive Command
   ↓
Backup Location


10. Startup Process

Runs during startup and recovery.

Responsibilities:

Read WAL
Replay transactions
Recover database

Example:

Crash
   ↓
Startup Process
   ↓
Replay WAL
   ↓
Consistent State

Oracle equivalent:

Instance Recovery


11. WAL Sender Process

Used in Streaming Replication.

Primary server:

Primary DB
   |
 WAL Sender
   |
 Network
   |
 WAL Receiver
   |
 Standby

Check:

SELECT *
FROM pg_stat_replication;


12. WAL Receiver Process

Runs on standby.

Purpose:

Receive WAL
from primary

Apply changes continuously.

Check:

SELECT *
FROM pg_stat_wal_receiver;


13. Logical Replication Launcher

Used for:

Logical Replication
Debezium
CDC
Kafka

Launches logical replication workers.

Example:

SELECT *
FROM pg_stat_subscription;


Shared Memory Interaction

All background processes interact with shared memory.

               Shared Memory
      ----------------------------------
      | Shared Buffers                |
      | WAL Buffers                   |
      | Lock Manager                  |
      | Proc Array                    |
      ----------------------------------
                  ^
                  |
    ------------------------------------
    |      |      |       |           |
 Backend WAL    BGW   Checkpoint  Vacuum


Crash Scenario Example

Suppose:

UPDATE accounts
SET balance=4000
WHERE id=1;

Step 1

Backend updates shared buffer.

Backend
  ↓
Shared Buffer

Step 2

Backend creates WAL record.

Backend
  ↓
WAL Buffer

Step 3

WAL Writer flushes WAL.

WAL Writer
  ↓
Disk

Step 4

Checkpointer eventually writes data page.

Checkpointer
   ↓
Data File

Step 5

Server crashes.

Startup Process:

Reads WAL
Replays Missing Changes

Database becomes consistent again.


PostgreSQL vs Oracle Background Processes

PostgreSQLOracle Equivalent
PostmasterPMON + Listener coordination
Backend ProcessDedicated Server Process
WAL WriterLGWR
CheckpointerCKPT
Background WriterDBWR
Startup ProcessSMON Recovery
AutovacuumNo direct equivalent
WAL SenderData Guard Redo Transport
WAL ReceiverManaged Recovery Process
ArchiverARCn

Most Common Interview Question

Explain PostgreSQL Process Architecture

Answer:

PostgreSQL uses a process-based architecture where the Postmaster acts as the parent process. For every client connection, Postmaster creates a dedicated backend process. Several background processes support database operations, including the WAL Writer for durability, Checkpointer for writing dirty pages to disk, Background Writer for smoothing I/O, Autovacuum Workers for MVCC cleanup and freeze operations, Archiver for WAL archiving, and WAL Sender/Receiver processes for streaming replication. Together they ensure performance, crash recovery, and data consistency.

About the Author : Anurag Pandey | Database cloud Architect | Oracle | SQL Server | PostgreSQL | AI and Data Platform Enthusiast

Anurag Pandey is a Database Architect with expertise in Oracle, PostgreSQL, SQL Server, Linux, and enterprise database platforms and specializes in database architecture, performance tuning, high availability, disaster recovery, cloud modernization, and operational excellence. With extensive experience in enterprise-scale environments, Anurag is passionate about leveraging AI, automation, and modern data platforms to improve reliability, scalability, and business outcomes. He actively contributes to technical communities through knowledge sharing, architecture guidance, and thought leadership in database and data engineering domains.

Follow more articles at

Visit Blog

What is Write-Ahead Logging (WAL) in PostgreSQL ?

 

Write-Ahead Logging (WAL) in PostgreSQL

Write-Ahead Logging (WAL) is PostgreSQL's mechanism for ensuring durability, crash recovery, and replication.

The core principle is:

Before changing the actual data page on disk, PostgreSQL first writes a record describing the change to the WAL.

This follows the ACID property of Durability.


Why WAL Is Needed

Imagine an account table:

UPDATE accounts
SET balance = balance - 1000
WHERE id = 1;

Suppose:

Balance = 5000

New balance:

4000

Now consider:

1. Data page updated in memory
2. Power failure
3. Data page not written to disk

Without WAL:

Transaction committed
Change lost

Database corruption or inconsistency can occur.


WAL Principle

Instead of immediately writing the table page:

Transaction
     |
     v
Create WAL Record
     |
     v
Flush WAL to disk
     |
     v
Commit Success
     |
     v
Data page written later

PostgreSQL guarantees:

WAL reaches disk first

Hence the name:

Write Ahead Log


Internal Flow

Suppose:

UPDATE employees
SET salary = 10000
WHERE id = 1;

Step 1

Data page loaded into shared buffers.

Buffer Cache
salary=8000


Step 2

Update performed in memory.

Buffer Cache
salary=10000

The actual disk page is still unchanged.


Step 3

Generate WAL record.

Example (simplified):

Table: employees
Block: 123
Offset: 5
Old Salary: 8000
New Salary: 10000


Step 4

Write WAL record into WAL buffer.

Shared Buffers
      +
WAL Buffers


Step 5

COMMIT

COMMIT;

Before PostgreSQL returns:

COMMIT successful

it flushes WAL to disk.

pg_wal/

directory.


Step 6

Actual table page written later.

Background Writer
Checkpoint

may write table pages minutes later.


Crash Recovery Example

Before Crash

Table:

salary=8000

Memory:

salary=10000

WAL:

UPDATE salary=10000

WAL already flushed.


Power Failure

Server crashes

Memory lost.


Startup Recovery

PostgreSQL reads WAL:

salary=10000

and replays the operation.

Result:

salary=10000

is restored.


WAL Records

WAL does not store full tables.

Instead it stores changes.

Example:

Insert tuple
Delete tuple
Update tuple
Create index
Drop table

Everything important is logged.


WAL Files

Location:

$PGDATA/pg_wal/

Example:

000000010000000000000001
000000010000000000000002
000000010000000000000003

Typically:

16 MB

per WAL segment.

Check:

SHOW wal_segment_size;


WAL and Checkpoints

Without checkpoints:

Recovery would replay years of WAL

Very slow.

PostgreSQL periodically performs checkpoints.


Checkpoint

A checkpoint writes dirty pages from memory to disk.

Shared Buffers
        ↓
Disk

and records:

Checkpoint Position

in WAL.


Recovery After Checkpoint

Instead of replaying all WAL:

Start from last checkpoint
Apply newer WAL

Much faster.


WAL and COMMIT

A common interview question:

Does COMMIT write table data?

No.

COMMIT writes WAL

Not necessarily table pages.

Table pages can be written later.

This makes PostgreSQL much faster.


WAL Buffers

Parameter:

SHOW wal_buffers;

Example:

16 MB

Purpose:

Temporary memory
for WAL records

before flushing.


WAL Levels

Check:

SHOW wal_level;

Possible values:

minimal
replica
logical


minimal

Least WAL generation.

Used for:

Standalone database


replica

Default in modern PostgreSQL.

Supports:

Streaming Replication
PITR
Recovery


logical

Supports:

Logical Replication
Debezium
CDC
Kafka

Captures enough information to reconstruct row-level changes.


WAL and Replication

Primary server:

Generates WAL

Standby server:

Reads WAL

Applies the same changes.

Architecture:

Primary
   |
   | WAL Stream
   v
Standby

This is PostgreSQL Streaming Replication.


WAL and Point-In-Time Recovery (PITR)

One of PostgreSQL's strongest features.

Example:

10:00 Good
11:00 Good
11:30 User drops table

Need recovery to:

11:29

Restore:

Base Backup
+
WAL Records

Replay until:

11:29

Table restored.


Important WAL Parameters

wal_level

SHOW wal_level;

Usually:

replica


max_wal_size

SHOW max_wal_size;

Example:

4GB
8GB
16GB

Controls WAL growth before checkpoint.


min_wal_size

Example:

1GB

WAL retained after checkpoints.


checkpoint_timeout

Default:

5 minutes

Maximum interval between checkpoints.


synchronous_commit

SHOW synchronous_commit;

Values:

on
off
remote_apply
remote_write


Performance Trade-off

ON

Safer.

OFF

Faster but may lose a few transactions during crash.


Oracle Comparison

Oracle also has a similar mechanism.

PostgreSQLOracle
WALRedo Logs
WAL WriterLGWR
CheckpointCheckpoint
Crash RecoveryInstance Recovery
PITRRMAN Recovery
WAL StreamingData Guard Redo Shipping

For Oracle DBAs:

PostgreSQL WAL is conceptually equivalent to Oracle Redo Logs.


Interview Answer

Write-Ahead Logging (WAL) is PostgreSQL's durability mechanism. Every database change is first recorded in the WAL and flushed to disk before the transaction commit is acknowledged. If the server crashes before data pages are written, PostgreSQL replays the WAL during recovery to restore committed transactions.

One-Line Summary

Data page later, WAL first. PostgreSQL always writes the change record to WAL before writing the actual table page, enabling crash recovery, replication, and point-in-time recovery.

About the Author : Anurag Pandey | Database cloud Architect | Oracle | SQL Server | PostgreSQL | AI and Data Platform Enthusiast

Anurag Pandey is a Database Architect with expertise in Oracle, PostgreSQL, SQL Server, Linux, and enterprise database platforms and specializes in database architecture, performance tuning, high availability, disaster recovery, cloud modernization, and operational excellence. With extensive experience in enterprise-scale environments, Anurag is passionate about leveraging AI, automation, and modern data platforms to improve reliability, scalability, and business outcomes. He actively contributes to technical communities through knowledge sharing, architecture guidance, and thought leadership in database and data engineering domains.

Follow more articles at

Visit Blog

VACUUM vs VACUUM FULL in PostgreSQL

 

VACUUM vs VACUUM FULL in PostgreSQL

A common misconception is:

"VACUUM removes dead tuples, so table size should shrink."

Wrong.

Regular VACUUM and VACUUM FULL have very different objectives.


Understanding the Problem

Suppose a table has grown to:

orders = 100 GB

Live data:

30 GB

Dead tuples:

70 GB

After many UPDATEs and DELETEs.


Regular VACUUM

VACUUM orders;

What it does

  1. Finds dead tuples.
  2. Marks their space as reusable.
  3. Updates visibility information.
  4. Helps index-only scans.
  5. Advances freeze information.

What it DOES NOT do

It does not return space to the operating system.


Example

Before:

Table Size = 100 GB
Page 1  Live rows
Page 2  Dead rows
Page 3  Live rows
Page 4  Dead rows

After VACUUM:

Page 1  Live rows
Page 2  Free space
Page 3  Live rows
Page 4  Free space

File size remains:

100 GB

But PostgreSQL can reuse that free space for future inserts/updates.


Analogy

Think of a hotel.

Before VACUUM:

100 rooms
70 empty
30 occupied

VACUUM:

Marks empty rooms available

Hotel building still has:

100 rooms

The building doesn't shrink.


VACUUM FULL

VACUUM FULL orders;

What it does

Instead of reusing space inside the file:

Creates a new compact copy
Copies only live rows
Drops old file


Example

Before:

100 GB table
30 GB live
70 GB dead

After:

30 GB table

Space is returned to:

Operating System

immediately.


Analogy

Using the hotel example:

Before:

100 rooms
30 occupied
70 empty

VACUUM FULL:

Build a new 30-room hotel
Move guests
Demolish old building

Now building size becomes:

30 rooms


Internal Architecture

Regular VACUUM

+------------------+
| Existing Table   |
+------------------+
         |
         +--> mark dead space reusable

No table rewrite.


VACUUM FULL

Old Table
     |
     v
Create New Table
     |
Copy Live Rows
     |
Swap Files
     |
Delete Old Table

Complete rewrite.


Locking Behavior

VACUUM

VACUUM table_name;

Lock:

SHARE UPDATE EXCLUSIVE

Practical impact:

✅ SELECT works

✅ INSERT works

✅ UPDATE works

✅ DELETE works

Usually invisible to users.


VACUUM FULL

VACUUM FULL table_name;

Lock:

ACCESS EXCLUSIVE

Impact:

❌ SELECT blocked

❌ INSERT blocked

❌ UPDATE blocked

❌ DELETE blocked

Everything waits.


Example:

500 GB table

VACUUM FULL might run:

Several hours

During which the table is unavailable.


Extra Disk Requirement

VACUUM

Needs very little additional disk.


VACUUM FULL

Needs enough space to hold:

Old table
+
New table

Example:

Current size = 500 GB

May require:

~500 GB extra

temporarily.


Index Impact

VACUUM

Indexes stay as-is.

Some index bloat may remain.


VACUUM FULL

Indexes are rebuilt.

Example:

Table bloat removed
Index bloat removed

Result:

Faster scans
Smaller indexes


Performance Comparison

FeatureVACUUMVACUUM FULL
Removes dead tuplesYesYes
Reclaims internal spaceYesYes
Returns disk to OSNoYes
Rewrites tableNoYes
Rebuilds indexesNoYes
Requires exclusive lockNoYes
AutoVacuum uses itYesNo
Suitable for production useYesRarely

Check Whether VACUUM Is Enough

Query:

SELECT
    relname,
    n_live_tup,
    n_dead_tup
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC;

If:

dead tuples present
but table size reasonable

Use:

VACUUM;


When VACUUM FULL Makes Sense

Case 1: Massive Delete

DELETE FROM orders
WHERE order_date < '2020-01-01';

Deleted:

90% of rows

Table remains:

1 TB

Live data:

100 GB

VACUUM won't shrink it.

VACUUM FULL can reduce:

1 TB -> 100 GB


Case 2: One-Time Data Purge

Archive and remove old data.

Then run:

VACUUM FULL;


Case 3: After Bulk Migration Cleanup

Drop old rows
Keep small dataset

Reclaim storage.


Why Production DBAs Prefer pg_repack

Instead of:

VACUUM FULL

many DBAs use:

pg_repack

Benefits:

✅ Removes bloat

✅ Rebuilds indexes

✅ Minimal locking

✅ Works online

Very common on:

Large OLTP systems
Financial databases
E-commerce databases


Interview Answer

VACUUM removes dead tuples and makes the space reusable inside the table file, but it does not reduce the file size on disk. It runs without blocking normal application activity and is what Autovacuum uses.

VACUUM FULL rewrites the entire table and indexes into a new compact copy, returning unused space to the operating system. However, it requires an ACCESS EXCLUSIVE lock, blocks all access to the table, and needs additional temporary disk space.

One-line summary:

VACUUM reuses free space inside the table; VACUUM FULL shrinks the table by rebuilding it.

About the Author : Anurag Pandey | Database cloud Architect | Oracle | SQL Server | PostgreSQL | AI and Data Platform Enthusiast

Anurag Pandey is a Database Architect with expertise in Oracle, PostgreSQL, SQL Server, Linux, and enterprise database platforms and specializes in database architecture, performance tuning, high availability, disaster recovery, cloud modernization, and operational excellence. With extensive experience in enterprise-scale environments, Anurag is passionate about leveraging AI, automation, and modern data platforms to improve reliability, scalability, and business outcomes. He actively contributes to technical communities through knowledge sharing, architecture guidance, and thought leadership in database and data engineering domains.

Follow more articles at

Visit Blog

What is Table Bloat in PostgreSQL Database ?

 

PostgreSQL Table Bloat Explained in Detail

Table bloat is one of the most important PostgreSQL performance issues.

Because PostgreSQL uses MVCC, every UPDATE creates a new row version instead of modifying the row in place.

How Bloat Happens

Assume a table:

CREATE TABLE employees(
   id INT,
   name TEXT
);

Initial data:

id   name
1    John


First Update

UPDATE employees
SET name='David'
WHERE id=1;

Internally:

Tuple 1: John    -> dead
Tuple 2: David   -> active

Physical storage:

Page 1
John   (dead)
David  (live)


More Updates

UPDATE employees SET name='Mike' WHERE id=1;
UPDATE employees SET name='Tom' WHERE id=1;
UPDATE employees SET name='James' WHERE id=1;

Now:

John    dead
David   dead
Mike    dead
Tom     dead
James   live

Although only one row exists logically, five versions exist physically.

80% dead tuples
20% live tuples

This is table bloat.


Why Bloat Is Bad

1. More Disk Usage

Logical data:

1 GB

Physical table:

20 GB

A common production issue.


2. More I/O

Suppose query:

SELECT *
FROM orders
WHERE order_id=100;

PostgreSQL scans pages.

Without bloat:

1000 pages

With bloat:

5000 pages

More disk reads.


3. Index Bloat

Every UPDATE creates new index entries.

Example:

Table Size = 50 GB
Indexes = 100 GB

Sometimes indexes become larger than the table.


4. Vacuum Takes Longer

More pages:

More work for VACUUM

which further increases maintenance cost.


Typical Causes

Long Running Transactions

Check:

SELECT pid,
       usename,
       now()-xact_start AS txn_age
FROM pg_stat_activity
WHERE xact_start IS NOT NULL
ORDER BY txn_age DESC;

Example:

3 hours transaction

Autovacuum cannot remove versions visible to that transaction.


Idle In Transaction Sessions

BEGIN;
-- user goes to lunch
COMMIT;

Very common application problem.

Check:

SELECT *
FROM pg_stat_activity
WHERE state='idle in transaction';


Heavy UPDATE Workloads

Examples:

Banking systems
Order systems
E-commerce inventory
Stock exchange data


Autovacuum Not Keeping Up

Autovacuum running slower than updates.

Result:

Dead tuples accumulate faster than cleanup.


Find Dead Tuples

Most useful first query:

SELECT
    schemaname,
    relname,
    n_live_tup,
    n_dead_tup,
    ROUND(
        100.0 * n_dead_tup /
        GREATEST(n_live_tup + n_dead_tup,1),
        2
    ) AS dead_pct
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC;

Example:

table         live       dead      dead_pct
------------------------------------------------
orders        1M         500K      33%
payments      2M         50K       2%

The orders table needs attention.


Find Largest Tables

SELECT
    relname,
    pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;

Example:

orders           500 GB
transactions     300 GB

Large tables with heavy updates are prime suspects.


Estimate Bloat Quickly

Top Suspects

SELECT
    relname,
    n_live_tup,
    n_dead_tup,
    pg_size_pretty(pg_total_relation_size(relid))
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC
LIMIT 20;


Exact Table Bloat Analysis

Install extension:

CREATE EXTENSION pgstattuple;

Check table:

SELECT *
FROM pgstattuple('orders');

Example output:

table_len         100 GB
tuple_len         40 GB
dead_tuple_len    45 GB
free_space        10 GB

Meaning:

45% table occupied by dead tuples

This is highly bloated.


Check Index Bloat

Install:

CREATE EXTENSION pgstattuple;

Check index:

SELECT *
FROM pgstatindex('idx_orders_order_id');

Example:

avg_leaf_density = 42%

Healthy:

>70%

Bloated:

<50%


Solution 1: Improve Autovacuum

For heavily updated tables:

ALTER TABLE orders
SET (
    autovacuum_vacuum_scale_factor = 0.01,
    autovacuum_analyze_scale_factor = 0.01
);

Instead of default:

20%

vacuum starts near:

1%

changes.


Solution 2: Manual VACUUM

VACUUM VERBOSE orders;

Benefits:

✅ Cleans dead tuples

✅ Updates visibility map

✅ Improves future index-only scans

Limitation:

❌ Does NOT shrink physical table size.


Solution 3: VACUUM FULL

VACUUM FULL orders;

What it does:

Rewrites complete table
Removes all dead space
Compacts table

Example:

Before: 100 GB
After : 35 GB


Downside

Exclusive lock.

Reads blocked
Writes blocked

until completion.

Not usually used on busy production systems.


Solution 4: pg_repack (Preferred)

Install:

pg_repack

Very popular production tool.

pg_repack -d mydb -t orders

Benefits:

✅ Rebuilds table

✅ Removes bloat

✅ Minimal locking

✅ Online

Example:

500 GB → 150 GB

without major downtime.

Most production DBAs prefer pg_repack over VACUUM FULL.


Find Tables Nearing Wraparound

Frequently correlated with poor vacuuming.

SELECT
    relname,
    age(relfrozenxid)
FROM pg_class
WHERE relkind='r'
ORDER BY age(relfrozenxid) DESC;

Large ages indicate vacuum is struggling.


Real Production Bloat Checklist

Run these in order:

Step 1

SELECT
    relname,
    n_live_tup,
    n_dead_tup
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC;

Step 2

SELECT *
FROM pg_stat_activity
WHERE state='idle in transaction';

Step 3

SELECT *
FROM pg_stat_progress_vacuum;

Step 4

SELECT *
FROM pgstattuple('large_table');

Step 5

If bloat > 30-40%:

Tune autovacuum
or
Run pg_repack


Interview Answer

PostgreSQL table bloat occurs because MVCC keeps old row versions after UPDATE or DELETE. When VACUUM cannot remove these dead tuples fast enough, tables and indexes grow larger than the actual live data. Bloat increases storage, I/O, query latency, and maintenance cost. It can be identified using pg_stat_user_tables, pgstattuple, and index statistics, and is typically resolved through proper autovacuum tuning, manual vacuuming, or online reorganization tools such as pg_repack.

About the Author : Anurag Pandey | Database cloud Architect | Oracle | SQL Server | PostgreSQL | AI and Data Platform Enthusiast

Anurag Pandey is a Database Architect with expertise in Oracle, PostgreSQL, SQL Server, Linux, and enterprise database platforms and specializes in database architecture, performance tuning, high availability, disaster recovery, cloud modernization, and operational excellence. With extensive experience in enterprise-scale environments, Anurag is passionate about leveraging AI, automation, and modern data platforms to improve reliability, scalability, and business outcomes. He actively contributes to technical communities through knowledge sharing, architecture guidance, and thought leadership in database and data engineering domains.

Follow more articles at

Visit Blog

PostgreSQL Process Architecture Diagram and Details

PostgreSQL Process Architecture Diagram                            +------------------+                            |    POSTMASTER    |     ...