How to analyze disk I/O performance in Linux

Use iostat and iotop to monitor real-time disk I/O, identify bottlenecks via %util and await, analyze process-level activity, inspect /proc/diskstats and /sys/block for system-wide stats, and benchmark with fio to measure performance under controlled workloads.

To analyze disk I/O performance in Linux, focus on identifying bottlenecks, monitoring active processes, and measuring throughput, latency, and utilization. Use built-in tools to get real-time and historical insights into how your storage subsystem behaves under load.

Monitor Real-Time Disk I/O with iostat

iostat, part of the sysstat package, shows detailed I/O statistics for devices and partitions.

• iostat -x 2 – Display extended statistics every 2 seconds
• Watch for high %util (device utilization) – consistently near 100% indicates a bottleneck
• Check await (average wait time in ms) – high values suggest slow storage or overload
• Compare svctm (service time) – though deprecated, it can hint at internal processing delays

Long await times with moderate utilization may point to inefficient I/O patterns or misconfigured storage.

Analyze Process-Level I/O Activity with iotop

iotop works like top but for I/O – it reveals which processes are reading/writing heavily.

• Run iotop (requires root or CAP_SYS_ADMIN)
• Sort by I/O rate using left/right arrow keys
• Look for processes with high DISK WRITE or READ values
• Identify if specific applications, databases, or background tasks (like backups) are causing spikes

This helps distinguish between expected heavy I/O and abnormal behavior.

Inspect Per-Device and System-Wide Stats via /proc and /sys

Linux exposes low-level I/O data through virtual filesystems.

• /proc/diskstats – Raw counters for reads, writes, and I/O time per device
• /sys/block/[device]/queue/ – Check queue depth, scheduler, and read_ahead_kb
• High I/O merge rates? Look at /sys/block/[dev]/statistics/*
• Use these when scripting custom monitoring or diagnosing kernel-level behavior

For instance, changing the I/O scheduler (none, deadline, mq-deadline, bfq) can impact performance on SSDs vs HDDs.

Generate Test Workloads with fio

To benchmark disk performance, use fio (flexible I/O tester).

• fio --name=randwrite --ioengine=libaio --rw=randwrite --bs=4k --numjobs=4 --size=1G --runtime=60 --time_based --direct=1 --output=fio-result.txt
• Adjust parameters for sequential/random, read/write, block size, and concurrency
• Measure IOPS, bandwidth (MB/s), and latency distributions
• Compare results across configurations (RAID, file systems, mount options)

Benchmarking helps validate hardware changes or tuning efforts.

Basically, combine observation tools like iostat and iotop with targeted testing using fio. Understand what’s happening now, then test what could be better. Most issues stem from overloaded drives, inefficient apps, or mismatched I/O settings.

The above is the detailed content of How to analyze disk I/O performance in Linux. For more information, please follow other related articles on the PHP Chinese website!