Locate and analyze the causes and consequences of memory leaks

Internal leak error code:

Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes)

Observe the memory usage of the php program

php provides two Method to get the memory usage of the current program.
memorygetusage(), the function of this function is to obtain the memory size currently used by the PHP script.

memorygetpeak_usage(), this function returns the peak memory occupied by the current script to the current position, so that it is possible to obtain the memory requirements of the current script.

int memory_get_usage ([ bool $real_usage = false ] )  
int memory_get_peak_usage ([ bool $real_usage = false ] )

The function gets by default the memory occupied by calling emalloc(). If the parameter is set to TRUE, the function gets the memory that the actual program applies to the system. Because PHP has its own memory management mechanism, sometimes although the memory has been released internally, it is not returned to the system.

The Linux system file /proc/{$pid}/status will record the running status of a process. The VmRSS field inside records the resident physical memory (Residence) used by the process. This is the actual process The physical memory occupied is more reliable. It is more reliable to use this data, and it is also easy to extract this value in the program.

Scenario 1: The program operation data is too large

Scenario restoration: Reading data that exceeds the upper limit of PHP’s available memory at one time leads to memory exhaustion

Example:

<?php  ini_set(&#39;memory_limit&#39;, &#39;128M&#39;);  
$string = str_pad(&#39;1&#39;, 128 * 1024 * 1024);    
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 134217729 bytes) 
in /Users/zouyi/php-oom/bigfile.php on line 3

This tells us that when the program is running and trying to allocate new memory, it throws a fatal error because it reaches the upper limit of memory allowed by PHP and cannot continue to execute. In Java development, it is generally called OOM. (Out Of Memory).
PHP configures the upper limit of memory by setting memory_limit in php.ini. The default value before PHP 5.2 was 8M, the default value for PHP 5.2 was 16M, and the default value for subsequent versions is 128M.
Problem phenomenon: It can be reproduced when processing specific data. You may encounter such problems when doing any IO operation, such as: returning a large amount of data in a mysql query, reading a large file into the program at a time, etc.

Solution:

1. Problems that can be solved with money are not problems. If there are not many opportunities for the program to read large files, and the upper limit is predictable, then through ini_set('memory_limit' , '1G'); to set a larger value or memory_limit=-1. If you have enough memory, you can keep the program running.

2. If the program needs to be able to be used normally on a small memory machine, then the program needs to be optimized. As shown below, the code is much more complicated.

<?php  
//php7 以下版本通过 composer 引入 paragonie/random_compat ，为了方便来生成一个随机名称的临时文件  
require "vendor/autoload.php";    
ini_set(&#39;memory_limit&#39;, &#39;128M&#39;);  
//生成临时文件存放大字符串  
$fileName = &#39;tmp&#39;.bin2hex(random_bytes(5)).&#39;.txt&#39;;  
touch($fileName);  
for ( $i = 0; $i < 128; $i++ ) {      
$string = str_pad(&#39;1&#39;, 1 * 1024 * 1024);      
file_put_contents($fileName, $string, FILE_APPEND);  
}  
$handle = fopen($fileName, "r");  
for ( $i = 0; $i <= filesize($fileName) / 1 * 1024 * 1024; $i++ )  {     
//do something     
$string = fread($handle, 1 * 1024 * 1024);  
}    
fclose($handle);  
unlink($fileName);

Scenario 2. Copies are generated when the program operates on big data

Scenario restoration: Large variables are copied during execution, resulting in insufficient memory.

<?php  
ini_set("memory_limit",&#39;1M&#39;);    
$string = str_pad(&#39;1&#39;, 1* 750 *1024);  
$string2 = $string;  $string2 .= &#39;1&#39;;    
Fatal error: Allowed memory size of 1048576 bytes exhausted (tried to allocate 768001 bytes) 
in /Users/zouyi/php-oom/unset.php on line 8    
Call Stack:      
0.0004     235440   1. {main}() /Users/zouyi/php-oom/unset.php:0    zend_mm_heap corrupted

Problem phenomenon: The memory occupied doubles during the execution of local code.

Problem analysis:
php is copy on write (Copy On Write), that is to say, the memory does not change when a new variable is assigned a value, and copying will not occur until the content of the new variable is manipulated. .

Solution:

Release useless variables early, or manipulate original data in the form of references.

<?php  
ini_set("memory_limit",&#39;1M&#39;);    
$string = str_pad(&#39;1&#39;, 1* 750 *1024);  
$string2 = $string;  unset($string);  
$string2 .= &#39;1&#39;;    
<?php  
ini_set("memory_limit",&#39;1M&#39;);    
$string = str_pad(&#39;1&#39;, 1* 750 *1024);  
$string2 = &$string;  
$string2 .= &#39;1&#39;;    
unset($string2, $string);

Scenario 3: System resources are exhausted due to unreasonable configuration

Scenario restoration: insufficient memory due to unreasonable configuration, the maximum setting on a 2G memory machine can be started at 100 php-fpm child processes, but after actually starting 50 php-fpm child processes, no more processes can be started.

Problem phenomenon: The problem does not occur when the online business request volume is small. Once the request volume is large, some requests will fail to execute.

Problem analysis: Generally for security reasons, PHP limits the maximum number and size of form requests that can be submitted and other parameters, post_max_size, max_file_uploads, upload_max_filesize, max_input_vars, max_input_nesting_level. Assuming that the bandwidth is sufficient, users frequently submit post_max_size = 8M data to the server, and nginx forwards it to php-fpm for processing, then in addition to the memory occupied by itself, each php-fpm child process may occupy 8M more memory even if it does nothing. .

Solution: Set parameters such as post_max_size, max_file_uploads, upload_max_filesize, max_input_vars, max_input_nesting_level appropriately and tune php-fpm related parameters.

php.ini code：

$ php -i |grep memory  
memory_limit => 1024M => 1024M //php脚本执行最大可使用内存  
$php -i |grep max  max_execution_time => 0 => 0 //最大执行时间，脚本默认为0不限制，web请求默认30s  
max_file_uploads => 20 => 20 //一个表单里最大上传文件数量  
max_input_nesting_level => 64 => 64 //一个表单里数据最大数组深度层数  
max_input_time => -1 => -1 //php从接收请求开始处理数据后的超时时间  
max_input_vars => 1000 => 1000 //一个表单（包括get、post、cookie的所有数据）最多提交1000个字段  
post_max_size => 8M => 8M //一次post请求最多提交8M数据  
upload_max_filesize => 2M => 2M //一个可上传的文件最大不超过2M

If the upload settings are unreasonable, it is not surprising that a large amount of memory is occupied. For example, in some intranet scenarios, the post needs to be extremely large. The string post_max_size=200M, then when 200M data is submitted from the form to the server, PHP will allocate 200M memory to this data until the request is processed and the memory is released.

Php-fpm.conf code:

pm = dynamic //仅dynamic模式下以下参数生效  
pm.max_children = 10 //最大子进程数  
pm.start_servers = 3 //启动时启动子进程数  
pm.min_spare_servers = 2 //最小空闲进程数，不够了启动更多进程  
pm.max_spare_servers = 5 //最大空闲进程数，超过了结束一些进程  
pm.max_requests = 500 //最大请求数，注意这个参数是一个php-fpm如果处理了500个请求后会自己重启一下，
可以避免一些三方扩展的内存泄露问题

One php-fpm process is calculated as 30MB of memory, and 50 php-fpm processes require 1500MB of memory. The requirements here are simple Estimate whether all php-fpm processes will run out of system memory under heaviest load.

Ulimit code:

$ulimit -a
-t: cpu time (seconds)              unlimited  
-f: file size (blocks)              unlimited  
-d: data seg size (kbytes)          unlimited  
-s: stack size (kbytes)             8192  
-c: core file size (blocks)         0  
-v: address space (kbytes)          unlimited  
-l: locked-in-memory size (kbytes)  unlimited  
-u: processes                       1024  
-n: file descriptors                1024

This is the configuration of my local mac os. The file descriptor setting is relatively small, and the general production environment configuration is much larger.

Scenario 4. Useless data is not released in time

情景还原：这种问题从程序逻辑上不是问题，但是无用的数据大量占用内存导致资源不够用，应该有针对性的做代码优化。 

Laravel开发中用于监听数据库操作时有如下代码: 

代码：

DB::listen(function ($query) {      
// $query->sql      
// $query->bindings      
// $query->time  
});

启用数据库监听后，每当有 SQL 执行时会 new 一个 QueryExecuted 对象并传入匿名函数以便后续操作，对于执行完毕就结束进程释放资源的php程序来说没有什么问题，而如果是一个常驻进程的程序，程序每执行一条 SQL 内存中就会增加一个 QueryExecuted 对象，程序不结束内存就会始终增长。

问题现象：程序运行期间内存逐渐增长，程序结束后内存正常释放。 

问题分析：此类问题不易察觉，定位困难，尤其是有些框架封装好的方法，要明确其适用场景。 

解决方法：本例中要通过DB::listen方法获取所有执行的 SQL 语句记录并写入日志，但此方法存在内存泄露问题，在开发环境下无所谓，在生产环境下则应停用，改用其他途径获取执行的 SQL 语句并写日志。

深入了解

1、名词解释

内存泄漏（Memory Leak）：是程序在管理内存分配过程中未能正确的释放不再使用的内存导致资源被大量占用的一种问题。在面向对象编程时，造成内存泄露的原因常常是对象在内存中存储但是运行中的代码却无法访问他。由于产生类似问题的情况很多，所以只能从源码上入手分析定位并解决。

垃圾回收（Garbage Collection，简称GC）：是一种自动内存管理的形式，GC程序检查并处理程序中那些已经分配出去但却不再被对象使用的内存。最早的GC是1959年前后John McCarthy发明的，用来简化在Lisp中手动控制内存管理。 PHP的内核中已自带内存管理的功能，一般应用场景下，不易出现内存泄露。

追踪法（Tracing）：从某个根对象开始追踪，检查哪些对象可访问，那么其他的（不可访问）就是垃圾。

引用计数法（reference count）：每个对象都一个数字用来标示被引用的次数。引用次数为0的可以回收。当对一个对象的引用创建时他的引用计数就会增加，引用销毁时计数减少。引用计数法可以保证对象一旦不被引用时第一时间销毁。但是引用计数有一些缺陷：1.循环引用，2.引用计数需要申请更多内存，3.对速度有影响，4.需要保证原子性，5.不是实时的。

2、php内存管理

在 PHP 5.3 以后引入了同步周期回收算法(Concurrent Cycle Collection)来处理内存泄露问题，代价是对性能有一定影响，不过一般 web 脚本应用程序影响很小。PHP的垃圾回收机制是默认打开的，php.ini 可以设置zend.enable_gc=0来关闭。也能通过分别调用gcenable() 和 gcdisable()函数来打开和关闭垃圾回收机制。 
虽然垃圾回收让php开发者在内存管理上无需担心了，但也有极端的反例：php界著名的包管理工具composer曾因加入一行gc_disable();性能得到极大提升。

3、php-fpm内存泄漏问题

在一台常见的 nginx + php-fpm 的服务器上： 
nginx 服务器 fork 出 n 个子进程（worker）， php-fpm 管理器 fork 出 n 个子进程。

当有用户请求， nginx 的一个 worker 接收请求，并将请求抛到 socket 中。

php-fpm 空闲的子进程监听到 socket 中有请求，接收并处理请求。

一个 php-fpm 的生命周期大致是这样的： 

模块初始化（MINIT）-> 请求初始化（RINIT）-> 请求处理 -> 请求结束（RSHUTDOWN） -> 请求初始化（RINIT）-> 请求处理 -> 请求结束（RSHUTDOWN）……. 请求初始化（RINIT）-> 请求处理 -> 请求结束（RSHUTDOWN）-> 模块关闭（MSHUTDOWN）。 

在请求初始化（RINIT）-> 请求处理 -> 请求结束（RSHUTDOWN）这个“请求处理”过程是： php 读取相应的 php 文件，对其进行词法分析，生成 opcode ， zend 虚拟机执行 opcode 。 
php 在每次请求结束后自动释放内存，有效避免了常见场景下内存泄露的问题，然而实际环境中因某些扩展的内存管理没有做好或者 php 代码中出现循环引用导致未能正常释放不用的资源。 
在 php-fpm 配置文件中，将pm.max_requests这个参数设置小一点。这个参数的含义是：一个 php-fpm 子进程最多处理pm.max_requests个用户请求后，就会被销毁。当一个 php-fpm 进程被销毁后，它所占用的所有内存都会被回收。 

4、常驻进程内存泄漏问题

Valgrind 包括如下一些工具： 
Memcheck。这是 valgrind 应用最广泛的工具，一个重量级的内存检查器，能够发现开发中绝大多数内存错误使用情况，比如：使用未初始化的内存，使用已经释放了的内存，内存访问越界等。

Callgrind。它主要用来检查程序中函数调用过程中出现的问题。

Cachegrind。它主要用来检查程序中缓存使用出现的问题。

Helgrind。它主要用来检查多线程程序中出现的竞争问题。

Massif。它主要用来检查程序中堆栈使用中出现的问题。

Extension。可以利用core提供的功能，自己编写特定的内存调试工具。

Memcheck 对调试 C/C++ 程序的内存泄露很有帮助，它的机制是在系统 alloc/free 等函数调用上加计数。 php 程序的内存泄露，是由于一些循环引用，或者 gc 的逻辑错误, valgrind 无法探测，因此需要在检测时需要关闭 php 自带的内存管理。 

代码：

$ export USE_ZEND_ALLOC=0   
# 设置环境变量关闭内存管理  
 valgrind --tool=memcheck --num-callers=30 --log-file=php.log
/Users/zouyi/Downloads/php-5.6.31/sapi/cli/php  leak.php

引用：

definitely lost: 肯定内存泄露 
indirectly lost: 非直接内存泄露 
possibly lost: 可能发生内存泄露 
still reachable: 仍然可访问的内存 
suppressed: 外部造成的内存泄露

Callgrind 配合 php 扩展 xdebug 输出的 profile 分析日志文件可以分析程序运行期间各个函数调用时占用的内存、 CPU 占用情况。 

总结：遇到了内存泄露时先观察是程序本身内存不足还是外部资源导致，然后搞清楚程序运行中用到了哪些资源：写入磁盘日志、连接数据库 SQL 查询、发送 Curl 请求、 Socket 通信等， I/O 操作必然会用到内存，如果这些地方都没有发生明显的内存泄露，检查哪里处理大量数据没有及时释放资源，如果是 php 5.3 以下版本还需考虑循环引用的问题。多了解一些 Linux 下的分析辅助工具，解决问题时可以事半功倍。 
最后宣传一下穿云团队今年最新开源的应用透明链路追踪工具 Molten：https://github.com/chuan-yun/Molten。安装好php扩展后就能帮你实时收集程序的 curl,pdo,mysqli,redis,mongodb,memcached 等请求的数据，可以很方便的与 zipkin 集成。 

以上内容仅供参考！

The above is the detailed content of Locate and analyze the causes and consequences of memory leaks. For more information, please follow other related articles on the PHP Chinese website!