Fast telemetry, hassle-free Linux performance analysis

As an operation and maintenance personnel, have you ever been troubled by performance problems of Linux systems? Maybe you have spent a lot of time and energy troubleshooting the problem, but still cannot accurately locate and analyze the problem. If so, then you need to know some ways to quickly telemetry Linux performance issues.

By running the following ten commands, you can roughly understand the running processes and resource usage of the system within sixty seconds. By looking at the error messages and resource saturation output from these commands (which are easy to understand), you can then optimize your resources. Saturation is when the load on a resource exceeds the limit it can handle. Once saturation occurs, it is usually exposed in the length or wait time of the request queue.

uptime
dmesg | tail
vmstat 1
mpstat -P ALL 1
pidstat 1
iostat -xz 1
free -m
sar -n DEV 1
sar -n TCP,ETCP 1
top

Some of these commands require pre-installation of the sysstat software package. The information displayed by these commands can help you implement the USE method (a method for locating performance bottlenecks), such as checking various resources (such as CPU, memory, disk etc.) usage, saturation and error information. In addition, during the process of locating the problem, you can use these commands to rule out certain possibilities that cause the problem, help you narrow the scope of inspection, and indicate the direction for the next step of inspection. Below The chapters will use the execution of these commands in a production environment as an example to briefly introduce these commands. If you want to learn more about the use of these tools, please refer to their man documents.

1. uptime

$ uptime
23:51:26 up 21:31, 1 user, load average: 30.02, 26.43, 19.02

This is a way to quickly check the system load average. It shows how many tasks (processes) there are to run in the system. In Linux systems, these numbers include the processes that need to run in the CPU and the number of processes that are running. Processes waiting for I/O (usually disk I/O), it is only a rough display of the system load, just look at it, you will need to use other tools to further understand the specific situation.
The last three numbers show the result of exponentially compressing the total load average of the system over one, five, and fifteen minutes. We can see how the system load changes over time. In the above example, the system load is increasing over time because the load value in the last minute exceeded 30, while the 15-minute average load was only 19. Such an obvious difference has many implications, such as CPU load. For further confirmation, run the vmstat or mpstat command. Please refer to Chapters 3 and 4 for these two commands.

2. dmesg | tail

$ dmesg | tail[1880957.563150] perl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
[...]
[1880957.563400] Out of memory: Kill process 18694 (perl) score 246 or sacrifice child
[1880957.563408] Killed process 18694 (perl) total-vm:1972392kB, anon-rss:1953348kB, file-rss:0kB
[2320864.954447] TCP: Possible SYN flooding on port 7001. Dropping request. Check SNMP counters.

This command displays the last 10 system messages, of course, provided they still exist, looking for errors that can cause performance problems.
The above example includes oom-killer, and TCP dropping a request, don't miss this step! The dmesg command is always worth a try.

3. vmstat 1

$ vmstat 1procs ---------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
34 0 0 200889792 73708 591828 0 0 0 5 6 10 96 1 3 0 0
32 0 0 200889920 73708 591860 0 0 0 592 13284 4282 98 1 1 0 0
32 0 0 200890112 73708 591860 0 0 0 0 9501 2154 99 1 0 0 0
32 0 0 200889568 73712 591856 0 0 0 48 11900 2459 99 0 0 0 0
32 0 0 200890208 73712 591860 0 0 0 0 15898 4840 98 1 1 0 0
^C

vmstat(8) is the abbreviation of virtual memory statistics. It is a commonly used tool (created for BSD decades ago). It will print a statistical summary of key servers at the end of each line. The vmstat command specifies a parameter. 1 Run is to print the statistical summary of each second. (This version of vmstat in the example) The columns in the first line of output are explicitly the average value since booting, not the value of the previous second.
For now, let's skip the first row, unless you want to know and remember each column, check these columns:

r：CPU 中正在运行和等待运行的进程的数量，其提供了一个比平均负载更好的信号来确定 CPU 是否饱和，因为其不包含 I/O，如果“r”的值大于了 CPU 的数量就表示已经饱和了。
free：以 kb 为单位显式的空闲内存,如果数字位数很多，说明你有足够的空闲内存,“free -m” 命令，是下面的第七个命令，其可以更好的说明空闲内存的状态。
si, so：代表Swap-ins 和 swap-outs,如果它们不是零，则代表你的内存不足了。
us, sy, id, wa, st：这些都是平均了所有 CPU 的 CPU 分解时间，它们分别是用户时间（user）、系统时间（内核）（system）、空闲（idle）、等待 I/O（wait）、以及占用时间（stolen）（被其他访客，或使用 Xen，访客自己独立的驱动域）。

CPU decomposition time will be determined by user time plus system time to confirm whether the CPU is busy; if the waiting time for I/O remains unchanged, it indicates a disk bottleneck; this is the idleness of the CPU, because tasks are blocked waiting to be suspended. When disk I/O is started, you can think of waiting for I/O as another form of CPU idle. It gives a clue as to why the CPU is idle. System time is very important for I/O processing. , an average system time above 20% may warrant further exploration: perhaps the kernel is too inefficient in handling I/O.
In the above example, the CPU time is spent almost entirely at the user level, indicating that the application is taking up too much CPU time, and the average CPU usage is also above 90%, of course this is not necessarily a problem, check "r "You can tell by the saturation in the column.

4. mpstat -P ALL 1

$ mpstat -P ALL 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)

07:38:49 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:38:50 PM all 98.47 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 0.78
07:38:50 PM 0 96.04 0.00 2.97 0.00 0.00 0.00 0.00 0.00 0.00 0.99
07:38:50 PM 1 97.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00
07:38:50 PM 2 98.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00
07:38:50 PM 3 96.97 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.03
[...]

This command prints the CPU breakdown time of each CPU, which can be used to check for an uneven usage. A single CPU that is busy represents a single-threaded application running.

5. pidstat 1

$ pidstat 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)

07:41:02 PM UID PID %usr %system %guest %CPU CPU Command
07:41:03 PM 0 9 0.00 0.94 0.00 0.94 1 rcuos/0
07:41:03 PM 0 4214 5.66 5.66 0.00 11.32 15 mesos-slave
07:41:03 PM 0 4354 0.94 0.94 0.00 1.89 8 java
07:41:03 PM 0 6521 1596.23 1.89 0.00 1598.11 27 java
07:41:03 PM 0 6564 1571.70 7.55 0.00 1579.25 28 java
07:41:03 PM 60004 60154 0.94 4.72 0.00 5.66 9 pidstat

07:41:03 PM UID PID %usr %system %guest %CPU CPU Command
07:41:04 PM 0 4214 6.00 2.00 0.00 8.00 15 mesos-slave
07:41:04 PM 0 6521 1590.00 1.00 0.00 1591.00 27 java
07:41:04 PM 0 6564 1573.00 10.00 0.00 1583.00 28 java
07:41:04 PM 108 6718 1.00 0.00 0.00 1.00 0 snmp-pass
07:41:04 PM 60004 60154 1.00 4.00 0.00 5.00 9 pidstat
^C

The pidstat command is a bit like the top command's statistical summary of each process, but it prints a rolling statistical summary in a loop instead of top's screen refresh. It can be used for real-time viewing, and it can also copy what you see (copy Paste) into your investigation record.
The above example shows that two Java processes are consuming the CPU. The %CPU column is the total of all CPUs; 1591% means that this Java process consumes nearly 16 CPUs.

6. iostat -xz 1

$ iostat -xz 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
73.96 0.00 3.73 0.03 0.06 22.21

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvda 0.00 0.23 0.21 0.18 4.52 2.08 34.37 0.00 9.98 13.80 5.42 2.44 0.09
xvdb 0.01 0.00 1.02 8.94 127.97 598.53 145.79 0.00 0.43 1.78 0.28 0.25 0.25
xvdc 0.01 0.00 1.02 8.86 127.79 595.94 146.50 0.00 0.45 1.82 0.30 0.27 0.26
dm-0 0.00 0.00 0.69 2.32 10.47 31.69 28.01 0.01 3.23 0.71 3.98 0.13 0.04
dm-1 0.00 0.00 0.00 0.94 0.01 3.78 8.00 0.33 345.84 0.04 346.81 0.01 0.00
dm-2 0.00 0.00 0.09 0.07 1.35 0.36 22.50 0.00 2.55 0.23 5.62 1.78 0.03
[...]
^C

无论是对工作负载还是性能表现来说，这都是一个很棒的用于查看块设备（磁盘）情况的工具。
查看个列：

r/s, w/s, rkB/s, wkB/s：这些分别代表该设备每秒的读次数、写次数、读取 kb 数，和写入 kb 数，这些数值用于描述工作负载情况，性能迟滞问题则可能仅仅是由于施加了过大的负载。
await：表示以毫秒为单位的 I/O 平均消耗时间，这是应用程序消耗的实际时间，因为它包括了排队时间和处理时间，比预期耗用了更多的平均时间就可能意味着设备的饱和，或设备出了问题。
avgqu-sz：向设备发出的请求的平均数量，该值大于 1 说明已经饱和了（虽说设备可以并行处理请求，尤其是由多个磁盘组成的虚拟设备。）
%util：设备利用率，这个值是一个显示出该设备在工作时每秒处于忙碌状态的百分比，虽然它取决于设备本身物理性能，若值大于 60％，通常表明性能不佳（可以从 await 中看出），当值接近 100% 通常意味着已饱和。

如果该存储设备是一个面向很多后端磁盘的逻辑磁盘设备，则 100% 利用率可能只是意味着当前正在处理某些 I/O 占用，然而，后端磁盘可能远未饱和，并且可能能够处理更多的工作，请记住，磁盘 I/O 性能较差不一定是程序的问题，许多技术通常是异步 I/O，使应用程序不会被阻塞并遭受延迟（例如，预读，以及写缓冲）。

7. free -m

$ free -m
total used free shared buffers cached
Mem: 245998 24545 221453 83 59 541
-/+ buffers/cache: 23944 222053
Swap: 0 0 0

右边的两列显式：

buffers：用于块设备 I/O 的缓冲区缓存。
cached：用于文件系统的页面缓存。

我们只是想要检查这些不接近零的大小，其可能会导致更高磁盘 I/O（使用 iostat 确认），和更糟糕的性能，上面的例子看起来还不错，每一列均有很多 M 个大小，比起第一行，-/+ buffers/cache 提供的内存使用量会更加准确些，Linux 会把暂时用不上的内存用作缓存，一旦应用需要的时候就立刻重新分配给它，所以部分被用作缓存的内存其实也算是空闲的内存，为了解释这一点， 甚至有人专门建了个网站： linuxatemyram，您如果还是不理解的话，可以去“充充电”。如果你在 Linux 上安装了 ZFS，这一点会变得更加困惑，因为 ZFS 它自己的文件系统缓存不算入free -m，有时候发现系统已经没有多少空闲内存可用了，其实内存却都待在 ZFS 的缓存里。

8. sar -n DEV 1

$ sar -n DEV 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)

12:16:48 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
12:16:49 AM eth0 18763.00 5032.00 20686.42 478.30 0.00 0.00 0.00 0.00
12:16:49 AM lo 14.00 14.00 1.36 1.36 0.00 0.00 0.00 0.00
12:16:49 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:16:49 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
12:16:50 AM eth0 19763.00 5101.00 21999.10 482.56 0.00 0.00 0.00 0.00
12:16:50 AM lo 20.00 20.00 3.25 3.25 0.00 0.00 0.00 0.00
12:16:50 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
^C

这个工具可以被用来检查网络接口的吞吐量：rxkB/s 和 txkB/s，以及是否达到限额，上面的例子中，eth0 接收的流量达到 22Mbytes/s，也即 176Mbits/sec（限额是 1Gbit/sec），我们所用的命令版本中还提供了 %ifutil 作为设备使用率（接收和发送的最大值）的指标，我们也可以用 Brendan 的 nicstat 工具计量这个值，一如 nicstat，sar 显示的这个值是很难精确取得的，在这个例子里面，它就没在正常的工作（显示0.00）。

9. sar -n TCP,ETCP 1

$ sar -n TCP,ETCP 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)

12:17:19 AM active/s passive/s iseg/s oseg/s
12:17:20 AM 1.00 0.00 10233.00 18846.00

12:17:19 AM atmptf/s estres/s retrans/s isegerr/s orsts/s
12:17:20 AM 0.00 0.00 0.00 0.00 0.00

12:17:20 AM active/s passive/s iseg/s oseg/s
12:17:21 AM 1.00 0.00 8359.00 6039.00

12:17:20 AM atmptf/s estres/s retrans/s isegerr/s orsts/s
12:17:21 AM 0.00 0.00 0.00 0.00 0.00
^C

这是一些关键的 TCP 指标的汇总视图，这些包括：

active/s：每秒本地发起 TCP 连接数（例如，通过 connect()）。
passive/s：每秒远程发起的 TCP 连接数（例如，通过 accept()）。
retrans/s：每秒重传 TCP 次数。

active 和 passive 的连接数往往对于描述一个粗略衡量服务器负载是非常有用的：新接受的连接数（passive），下行连接数（active），也可以理解为 active 连接是对外的，而 passive 连接是对内的，虽然严格来说并不完全正确（例如，一个 localhost 到 localhost 的连接），重传是出现一个网络和服务器问题的一个征兆，其可能是由于一个不可靠的网络（例如，公网）造成的，或许也有可能是由于服务器过载并丢包。上面的例子显示了每秒只有一个新的 TCP 连接。

10. top

$ toptop - 00:15:40 up 21:56, 1 user, load average: 31.09, 29.87, 29.92
Tasks: 871 total, 1 running, 868 sleeping, 0 stopped, 2 zombie
%Cpu(s): 96.8 us, 0.4 sy, 0.0 ni, 2.7 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 25190241+total, 24921688 used, 22698073+free, 60448 buffers
KiB Swap: 0 total, 0 used, 0 free. 554208 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20248 root 20 0 0.227t 0.012t 18748 S 3090 5.2 29812:58 java
4213 root 20 0 2722544 64640 44232 S 23.5 0.0 233:35.37 mesos-slave
66128 titancl+ 20 0 24344 2332 1172 R 1.0 0.0 0:00.07 top
5235 root 20 0 38.227g 547004 49996 S 0.7 0.2 2:02.74 java
4299 root 20 0 20.015g 2.682g 16836 S 0.3 1.1 33:14.42 java
1 root 20 0 33620 2920 1496 S 0.0 0.0 0:03.82 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:05.35 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:06.94 kworker/u256:0
8 root 20 0 0 0 0 S 0.0 0.0 2:38.05 rcu_sched

top 命令包含了很多我们之前已经检查过的指标，我们可以观察到命令输出的结果每时每刻都有很大不同，这表明负载是可变的，top 的一个缺点是，很难看到数据随时间变动的趋势，而vmstat 和 pidstat则能提供滚动输出，这样显得更清楚一些，同样的，如果你不以足够快的速度暂停输出以上信息（Ctrl-S 暂停，Ctrl-Q 继续），一些间歇性问题的线索就可能由于被清屏而丢失。

综上所述，通过本文中介绍的快速遥测技巧，我们可以更加高效地诊断和解决Linux系统的性能问题。当出现问题时，我们可以快速地排查，在最短的时间内找到关键点并进行调试，从而提高工作效率和用户体验。

The above is the detailed content of Fast telemetry, hassle-free Linux performance analysis. For more information, please follow other related articles on the PHP Chinese website!