How to export large amounts of data in php

In the process of PHP development, sometimes there is data that needs to be exported. You can use some packages in various frameworks to directly obtain the data and then throw it into the corresponding processing method. However, when encountering a large amount of data that needs to be exported, At that time, we need to use other methods to achieve it.

Generally, there are several solutions to deal with big data: Generate data by adding or modifying event trigger scripts; generate data by executing scheduled task scripts; directly use phpexcel to export large amounts of data; use fputcsv real-time data flow to write . Use a picture to represent it as follows:

#Here we mainly record the use of fputcsv to export data.

public function exportData()
    {   
        set_time_limit(0);
        ini_set('memory_limit', '1024M');        
        $columns = [            '列名1', '列名2', '列名3'      //需要几列，定义好列名
        ];        //设置好告诉浏览器要下载excel文件的headers
        header('Content-Description: File Transfer');
        header('Content-Type: application/vnd.ms-excel');
        header('Content-Disposition: attachment; filename="导出数据-'.date('Y-m-d', time()).'.csv"');
        header('Expires: 0');
        header('Cache-Control: must-revalidate');
        header('Pragma: public');        $fp = fopen('php://output', 'a');//打开output流
        mb_convert_variables('GBK', 'UTF-8', $columns);
        fputcsv($fp, $columns);//将数据格式化为CSV格式并写入到output流中

        //添加查询条件，获取需要的数据
        $query = Model::class()->where();

        //获取总数，分页循环处理
        $accessNum = $query->count();
        $perSize = 1000;
        $pages   = ceil($accessNum / $perSize);        
        for($i = 1; $i <= $pages; $i++) {

            $db_data = $query->limit($perSize)->offset(($i-1)*$perSize)->get();            
            foreach($db_data as $key => $value) {                
            $rowData = [];                //获取每列数据，转换处理成需要导出的数据
                //需要格式转换，否则会乱码
                mb_convert_variables(&#39;GBK&#39;, &#39;UTF-8&#39;, $rowData);
                fputcsv($fp, $rowData);
            }            //释放变量的内存
            unset($db_data);            //刷新输出缓冲到浏览器
            ob_flush();            //必须同时使用 ob_flush() 和flush() 函数来刷新输出缓冲。
            flush();
        }

        fclose($fp);        
        exit();
    }

This will generate and write the downloaded file at the same time, improving the processing speed.
The package processing used before generates a 25M data, which takes 20 minutes. This method of processing takes more than 1 minute (also related to the actual processing environment).

Related recommendations:

PHP csv large amount of data export split processing

php import large amount of data to mysql (example)

The above is the detailed content of How to export large amounts of data in php. For more information, please follow other related articles on the PHP Chinese website!