How to monitor memory changes in the local environment and production environment in the Node service?

When usingNodeas a server language in a production environment, excessive concurrency or code problems cause OOM (out of memory) or the CPU is fully loaded. These are common problems in servers. At this time, it is easy to find problems by monitoring CPU and memory, combined with logs and Release.

[Video tutorial recommendation:node js tutorial]

This chapter will introduce how to monitor memory changes in the local environment and production environment

A Node application instance

So, how to dynamically monitor the memory changes of a Node process?

The following is an example of a Node Server, and it is an example with a memory leak problem, and it is a streamlined version of the problem that Shanyue has been positioning in the production environment for a long time.

In that memory leak problem, the memory in a single container skyrocketed from the original 400M to 700M. Under the 800M container resource limit, OOM would occasionally occur, causing a restart. The problem was not located for a while (the problem was discovered too late, the time series data half a month ago had been swallowed up, so Release was not located), so the resource limit was raised to 1000M. Later it was found that it was caused by ctx.request mounting a large field in the database

const Koa = require('koa') const app = new Koa() function getData () { return Array.from(Array(1000)).map(x => 10086) } app.use(async (ctx, next) => { ctx.data = getData() await next() }) app.use(ctx => { ctx.body = 'hello, world' }) app.listen(3200, () => console.log('Port: 3200'))

Process memory monitoring

Some problems need to be eliminated in time in the local and test environments , to avoid greater impact on the production environment. Then it is crucial to understand how to monitor memory locally.

pidstatis a package of thesysstatseries of Linux performance debugging tools. It is actually used to debug Linux performance problems, including memory, network, IO, CPU, etc.

This is not only tried withnode, but also applies to all processes, includingpython,javaandgo

# -r: 指输出内存指标 # -p: 指定 pid # 1: 每一秒输出一次 # 100: 输出100次 $ pidstat -r -p pid 1 100

Before usingpidstat, you need to find the pid of the processpid

How to find the pid of the Node process

Innodeyou can find the process’pid

> process.pid 16425

through

process.pid

although by writing The code can findpid, but is intrusive and not very practical. So how to findpidthrough non-invasive means? There are two ways

1. Combined with redundant parameterspsLocate the process
2. Combined with the port numberlsofLocate the process

$ node index.js shanyue # 第一种方法：通过多余的参数快速定位 pid $ ps -ef | grep shanyue root 31796 23839 1 16:38 pts/5 00:00:00 node index.js shanyue # 第二种方法：通过端口号定位 pid lsof -i:3200 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME node 31796 root 20u IPv6 235987334 0t0 TCP *:tick-port (LISTEN)

Use pidstat to monitor memory

From the above code, we can know that the pid of the node service is31796, in order to observe the dynamic changes of memory, apply a stress test

$ ab -c 10000 -n 1000000 http://localhost:3200/

# -r: 指输出内存指标 # -p: 指定 pid # 1: 每一秒输出一次 # 100: 输出100次 $ pidstat -r -p 31796 1 100 Linux 3.10.0-957.21.3.el7.x86_64 (shuifeng) 2020年07月02日 _x86_64_ (2 CPU) UID PID minflt/s majflt/s VSZ RSS %MEM Command 19时20分39秒 0 11401 0.00 0.00 566768 19800 0.12 node 19时20分40秒 0 11401 0.00 0.00 566768 19800 0.12 node 19时20分41秒 0 11401 9667.00 0.00 579024 37792 0.23 node 19时20分42秒 0 11401 11311.00 0.00 600716 59988 0.37 node 19时20分43秒 0 11401 5417.82 0.00 611420 70900 0.44 node 19时20分44秒 0 11401 3901.00 0.00 627292 85928 0.53 node 19时20分45秒 0 11401 1560.00 0.00 621660 81208 0.50 node 19时20分46秒 0 11401 2390.00 0.00 623964 83696 0.51 node 19时20分47秒 0 11401 1764.00 0.00 625500 85204 0.52 node

The meaning of the output indicators is as follows

• ##RSS:Resident Set Size, resident memory set, can be understood as memory, this is the memory indicator we need to monitor
• VSZ:virtual size, virtual memory

As can be seen from the output,

After the stress test was applied, the memory increased from 19M to 85M.

Use top to monitor memory

pidstatis a Linux performance tool undersysstat, but in mac, how to locate the memory Variety?

You can use

top/htop

$ htop -p 31796

Production environment memory monitoring

Since most of the current production environment Deployed in

k8s,So the memory monitoring of a certain application in the production environment is essentially the memory monitoring of a certainworkload/deploymentby k8s, about memory monitoring# The data flow direction of ##metricis roughly as follows:

k8s

->metric server->prometheus->grafanaThe architecture diagram is as follows:

##The above pictures are taken from the following articles

Kubernetes Monitoring with Prometheus

• Kubernetes monitoring architecture
Finally be able to A real-time graph of memory monitoring of an application collected in #grafana

:

Since there is too much design content in this part, I will introduce it in the following chapters

This is not only applicable to node services, but also applicable to all k8sworkload

Summary

This chapter introduces the monitoring of Node service memory in the local environment and production environment

1. Local usehtop/toporpidstatMonitor process memory

2. Use in production environmentk8s/metric-server/prometheus/grafanaMonitor the memory of the entire node application

When a memory leak is detected in a certain service, how to solve the problem? Therefore, the following article will talk about

1. How the production environment monitors the memory of the entire application

2. When OOM occurs in the production environment, how to quickly locate

3. Several examples of OOM positioning in real production environments

For more programming-related knowledge, please visit:Introduction to Programming! !

The above is the detailed content of How to monitor memory changes in the local environment and production environment in the Node service?. For more information, please follow other related articles on the PHP Chinese website!