Solution to memory exhaustion problem when looping large amounts of data in PHP_PHP Tutorial

Solution to memory exhaustion problem when PHP loops large amounts of data

I recently encountered the following error when developing a PHP program:

PHP Fatal error: Allowed memory size of 268 435 456 bytes exhausted

The error message shows that the maximum allowed memory has been exhausted. I was surprised to encounter such an error at first, but after thinking about it, it is not surprising, because the program I am developing is to use a foreach loop statement to search the entire table in a table with 40,000 records. For data with specific characteristics, that is to say, 40,000 pieces of data must be taken out at a time, and then the daily data must be checked one by one. It is conceivable that if all 40,000 pieces of data are loaded into the memory, it would be strange if the memory does not burst.

After all these years of programming, I vaguely remember that PHP provides an API that does not load data all at once. It is a query method that can be used and lost like streaming media, and the data does not accumulate in the memory. After a simple search, I found the correct usage on the official website.

This problem is called Buffered and Unbuffered queries on PHP’s official website. PHP's default query mode is buffered mode. In other words, the query data results will be extracted into memory all at once for processing by the PHP program. This gives the PHP program additional functions, such as counting the number of rows, pointing the pointer to a certain row, etc. What's more important is that the program can repeatedly perform secondary queries and filtering operations on the data set. However, the disadvantage of this buffered query mode is that it consumes memory, that is, it trades space for speed.

In contrast, another PHP query mode is a non-buffered query. The database server will return data one by one instead of all at once. The result is that the PHP program consumes less memory, but it increases the load of the database server. Pressure, because the database will keep waiting for PHP to fetch the data until all the data is fetched.

Obviously, the buffered query mode is suitable for queries with small data volumes, while non-buffered queries are suitable for queries with large data volumes.

Everyone knows about PHP’s buffered mode query. The example listed below is how to execute the non-buffered query API.

Non-buffered query method one: mysqli

$mysqli = new mysqli("localhost", "my_user", "my_password", "world");
$uresult = $mysqli->query("SELECT Name FROM City", MYSQLI_USE_RESULT);

if ($uresult) {
while ($row = $uresult->fetch_assoc()) {
echo $row['Name'] .PHP_EOL;
}
}
$uresult->close();
?>

Non-buffered query method two: pdo_mysql

$pdo = new PDO("mysql:host=localhost;dbname=world", 'my_user', 'my_pass');
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);

$uresult = $pdo->query("SELECT Name FROM City");
if ($uresult) {
while ($row = $uresult->fetch(PDO::FETCH_ASSOC)) {
echo $row['Name'] .PHP_EOL;
}
}
?>

Non-buffered query method three: mysql

$conn = mysql_connect("localhost", "my_user", "my_pass");
$db = mysql_select_db("world");

$uresult = mysql_unbuffered_query("SELECT Name FROM City");
if ($uresult) {
while ($row = mysql_fetch_assoc($uresult)) {
echo $row['Name'] .PHP_EOL;
}
}
?>