Use the sar tool to detect system performance bottlenecks

sarThe command is used to collect, report, or save UNIX/Linux system activity information. It saves selected counters to the operating system's/var/log/sa/saddfile. From the collected data, you can get a lot of information about your server:

1. CPU Usage
2. Memory pages and usage
3. Network I/O and Transfer Statistics
4. Process Creation Activity
5. All block device activities
6. Interruptions per second, etc.

sarThe output of the command can be used to identify server bottlenecks. However, it can be difficult to analyze the information provided by thesarcommand, so use the kSar tool. The kSar tool can plot the output of thesarcommand into an easy-to-understand chart based on time periods.

sar,sa1, andsa2commands are part of the sysstat package. It is a collection of performance monitoring tools included with Linux.

1. sar：Display data
2. sa1andsa2: Collect and save data for later analysis.sa2The shell script writes a daily report in the/var/log/sadirectory.sa1The shell script writes daily system activity information to a file in the form of binary data.
3. sadc - System activity data collector. You can configure various options by modifying thesa1andsa2scripts. They are located in the following directories:
   • /usr/lib64/sa/sa1(64-bit) or/usr/lib/sa/sa1(32-bit) - it callssadcGo to log reports to/var/log/sa/sadXformat.
   • /usr/lib64/sa/sa2(64-bit) or/usr/lib/sa/sa2(32-bit) - it callssarGo and log reports to/var/log/sa/sarXformat.

在一个基于 CentOS/RHEL 的系统上，输入如下的 yum 命令[1]去安装 sysstat：

# yum install sysstat

示例输出如下：

Loaded plugins: downloadonly, fastestmirror, priorities, : protectbase, security Loading mirror speeds from cached hostfile * addons: mirror.cs.vt.edu * base: mirror.ash.fastserv.com * epel: serverbeach1.fedoraproject.org * extras: mirror.cogentco.com * updates: centos.mirror.nac.net 0 packages excluded due to repository protections Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package sysstat.x86_64 0:7.0.2-3.el5 set to be updated --> Finished Dependency Resolution Dependencies Resolved ==================================================================== Package Arch Version Repository Size ==================================================================== Installing: sysstat x86_64 7.0.2-3.el5 base 173 k Transaction Summary ==================================================================== Install 1 Package(s) Update 0 Package(s) Remove 0 Package(s) Total download size: 173 k Is this ok [y/N]: y Downloading Packages: sysstat-7.0.2-3.el5.x86_64.rpm | 173 kB 00:00 Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Installing : sysstat 1/1 Installed: sysstat.x86_64 0:7.0.2-3.el5 Complete!

编辑/etc/sysconfig/sysstat文件去指定日志文件保存多少天（最长为一个月）：

# vi /etc/sysconfig/sysstat

示例输出如下 ：

# keep log for 28 days # the default is 7 HISTORY=28

保存并关闭这个文件。

默认的 cron 作业位于[2]/etc/cron.d/sysstat：

# cat /etc/cron.d/sysstat

示例输出如下：

# run system activity accounting tool every 10 minutes */10 * * * * root /usr/lib64/sa/sa1 1 1 # generate a daily summary of process accounting at 23:53 53 23 * * * root /usr/lib64/sa/sa2 -A

使用一个文本编辑器去编辑/etc/cron.d/sysstat文件，比如使用vim命令，输入如下：

# vi /etc/cron.d/sysstat

像下面的示例那样更新这个文件，以记录所有的硬盘统计数据（-d选项强制记录每个块设备的统计数据，而-I选项强制记录所有系统中断的统计数据）：

# run system activity accounting tool every 10 minutes */10 * * * * root /usr/lib64/sa/sa1 -I -d 1 1 # generate a daily summary of process accounting at 23:53 53 23 * * * root /usr/lib64/sa/sa2 -A

在 CentOS/RHEL 7.x 系统上你需要传递-S DISK选项去收集块设备的数据。传递-S XALL选项去采集如下所列的数据：

1. 磁盘
2. 分区
3. 系统中断
4. SNMP
5. IPv6

# Run system activity accounting tool every 10 minutes */10 * * * * root /usr/lib64/sa/sa1 -S DISK 1 1 # 0 * * * * root /usr/lib64/sa/sa1 600 6 & # Generate a daily summary of process accounting at 23:53 53 23 * * * root /usr/lib64/sa/sa2 -A # Run system activity accounting tool every 10 minutes

保存并关闭这个文件。

输入如下命令:

chkconfig sysstat on service sysstat start

示例输出如下：

Calling the system activity data collector (sadc):

对于 CentOS/RHEL 7.x，运行如下的命令：

# systemctl enable sysstat # systemctl start sysstat.service # systemctl status sysstat.service

示例输出：

● sysstat.service - Resets System Activity Logs Loaded: loaded (/usr/lib/systemd/system/sysstat.service; enabled; vendor preset: enabled) Active: active (exited) since Sat 2018-01-06 16:33:19 IST; 3s ago Process: 28297 ExecStart=/usr/lib64/sa/sa1 --boot (code=exited, status=0/SUCCESS) Main PID: 28297 (code=exited, status=0/SUCCESS) Jan 06 16:33:19 centos7-box systemd[1]: Starting Resets System Activity Logs... Jan 06 16:33:19 centos7-box systemd[1]: Started Resets System Activity Logs.

使用sar命令去显示操作系统中选定的累积活动计数器输出。在这个示例中，运行sar命令行，去实时获得 CPU 使用率的报告：

# sar -u 3 10

示例输出：

Linux 2.6.18-164.2.1.el5 (www-03.nixcraft.in) 12/14/2009 09:49:47 PM CPU %user %nice %system %iowait %steal %idle 09:49:50 PM all 5.66 0.00 1.22 0.04 0.00 93.08 09:49:53 PM all 12.29 0.00 1.93 0.04 0.00 85.74 09:49:56 PM all 9.30 0.00 1.61 0.00 0.00 89.10 09:49:59 PM all 10.86 0.00 1.51 0.04 0.00 87.58 09:50:02 PM all 14.21 0.00 3.27 0.04 0.00 82.47 09:50:05 PM all 13.98 0.00 4.04 0.04 0.00 81.93 09:50:08 PM all 6.60 6.89 1.26 0.00 0.00 85.25 09:50:11 PM all 7.25 0.00 1.55 0.04 0.00 91.15 09:50:14 PM all 6.61 0.00 1.09 0.00 0.00 92.31 09:50:17 PM all 5.71 0.00 0.96 0.00 0.00 93.33 Average: all 9.24 0.69 1.84 0.03 0.00 88.20

其中：

• 3 表示间隔时间
• 10 表示次数

查看进程创建的统计数据，输入：

# sar -c 3 10

查看 I/O 和传输率统计数据，输入：

# sar -b 3 10

查看内存页面统计数据，输入：

# sar -B 3 10

查看块设备统计数据，输入：

# sar -d 3 10

查看所有中断的统计数据，输入：

# sar -I XALL 3 10

查看网络设备特定的统计数据，输入：

# sar -n DEV 3 10 # sar -n EDEV 3 10

查看 CPU 特定的统计数据，输入：

# sar -P ALL # Only 1st CPU stats # sar -P 1 3 10

查看队列长度和平均负载的统计数据，输入：

# sar -q 3 10

查看内存和交换空间的使用统计数据，输入：

# sar -r 3 10 # sar -R 3 10

查看 inode、文件、和其它内核表统计数据状态，输入：

# sar -v 3 10

查看系统切换活动统计数据，输入：

# sar -w 3 10

查看交换统计数据，输入：

# sar -W 3 10

查看一个 PID 为 3256 的 Apache 进程，输入：

# sar -x 3256 3 10

sar和sadf提供了基于命令行界面的输出。这种输出可能会使新手用户/系统管理员感到无从下手。因此，你需要使用 kSar，它是一个图形化显示你的sar数据的 Java 应用程序。它也允许你以 PDF/JPG/PNG/CSV 格式导出数据。你可以用三种方式去加载数据：本地文件、运行本地命令、以及通过 SSH 远程运行的命令。kSar 可以处理下列操作系统的sar输出：

1. Solaris 8, 9 和 10
2. Mac OS/X 10.4+
3. Linux (Systat Version >= 5.0.5)
4. AIX (4.3 & 5.3)
5. HPUX 11.00+

访问 官方[3]网站去获得最新版本的源代码。使用 wget[4]去下载源代码，输入：

$ wget https://github.com/vlsi/ksar/releases/download/v5.2.4-snapshot-652bf16/ksar-5.2.4-SNAPSHOT-all.jar

首先要确保你的机器上 JAVA jdk[5]已安装并能够正常工作。输入下列命令去启动 kSar：

$ java -jar ksar-5.2.4-SNAPSHOT-all.jar

kSar welcome screen

接下来你将看到 kSar 的主窗口，和有两个菜单的面板。

kSar - the main window

左侧有一个列表，是 kSar 根据数据已经解析出的可用图表的列表。右侧窗口将展示你选定的图表。

首先，你需要从命名为 server1 的服务器上采集sar命令的统计数据。输入如下的命令：

[ server1 ]# LC_ALL=C sar -A > /tmp/sar.data.txt

接下来，使用scp命令从本地桌面拷贝到远程电脑上：

[ desktop ]$ scp user@server1.nixcraft.com:/tmp/sar.data.txt /tmp/

切换到 kSar 窗口，点击 “Data” > “Load data from text file” > 从/tmp/中选择sar.data.txt> 点击 “Open” 按钮。

现在，图表类型树已经出现在左侧面板中并选定了一个图形：

Linux kSar Processes for server1

Linux Disk I/O Stats Using kSar

Linux Memory paging and its utilization stats

放大和缩小
通过移动你可以交互式缩放图像的一部分。在要缩放的图像的左上角点击并按下鼠标，移动到要缩放区域的右下角，可以选定要缩放的区域。返回到未缩放状态，点击并拖动鼠标到除了右下角外的任意位置，你也可以点击并选择 zoom 选项。
了解 kSar 图像和 sar 数据
我强烈建议你去阅读sar和sadf命令的 man 页面：

$ man sar $ man sadf

使用sar命令和 kSar 工具，可以得到内存、CPU、以及其它子系统的详细快照。例如，如果 CPU 使用率在一个很长的时间内持续高于 80%，有可能就是出现了一个 CPU 瓶颈。使用sar -x ALL你可以找到大量消耗 CPU 的进程。

mpstat 命令[6]的输出（sysstat 包的一部分）也会帮你去了解 CPU 的使用率。但你可以使用 kSar 很容易地去分析这些信息。

Perform the following adjustments to the CPU:

1. Make sure there are no unnecessary processes running in the background. Turn off all unnecessary services on Linux[7].
2. Use cron[8]Run a task (for example, backup) at an off-peak time.
3. Use top and ps commands[9]to find all non-critical background jobs/services. Use the renice command[10]to adjust low priority jobs.
4. Use the taskset command to set the CPU used by the process[11](offload the CPU used), that is, bind the process to a different CPU. For example, running MySQL database on CPU #2 and Apache on CPU #3.
5. Make sure your system is using the latest drivers and firmware.
6. Add additional CPUs to the system if possible.
7. Use a faster CPU for single-threaded applications (e.g., Lighttpd web server application).
8. Use multiple CPUs for multi-threaded applications (e.g., MySQL database server applications).
9. Use multiple compute nodes and set up a load balancer for a web application[12].

isagcommand graphically displays system activity data stored in a binary file from previous runs of thesarcommand. Theisagcommand referencessarand extracts its data to plot the graph.isaghas fewer options compared to kSar.

Fig.06: isag CPU utilization graphs

The above is the detailed content of Use the sar tool to detect system performance bottlenecks. For more information, please follow other related articles on the PHP Chinese website!