Performance comparison of using file_get_content series functions and using curl series functions to collect images-PHP Tutorial-php.cn

Performance comparison of using file_get_content series functions and using curl series functions to collect images

WBOY

Release： 2016-08-08 09:27:44

Original

1066 people have browsed it

Since the car content in the backend of a company’s car website mainly comes from Auto Home, the editor colleagues have to manually add cars to Auto Home every day, which is really a pain in the ass. Ever since, in order to change this situation, as a development coder, my task has come. . . That is to prepare a function. As long as you paste the corresponding car home URL, the data can be automatically filled into the form in our backend. At present, the basic filling has been implemented, but the corresponding car photo album is still not collected. Come in.

I have done the function of collecting pictures before, but most of the cars in Autohome have a lot of pictures. At the beginning, I planned to use the previous method of collecting pictures, that is, use file_get_content to get the content corresponding to the URL. Then match the address of the image, then use file_get_content to obtain the content of these image URLs, and then load it locally. The code is as follows:

php
header('Content-type:text/html;charset=utf-8');
set_time_limit(0);

class runtime  
{  
    var $StartTime = 0;  
    var $StopTime = 0;  
   
    function get_microtime()  
    {  
        list($usec, $sec) = explode(' ', microtime());  
        return ((float)$usec + (float)$sec);  
    }  
   
    function start()  
    {  
        $this->StartTime = $this->get_microtime();  
    }  
   
    function stop()  
    {  
        $this->StopTime = $this->get_microtime();  
    }  
   
    function spent()  
    {  
        return round(($this->StopTime - $this->StartTime) * 1000, 1);  
    }  
   
}  

$runtime= new runtime();  
$runtime->start();  

$url = 'http://car.autohome.com.cn/pic/series-s15306/289.html#pvareaid=102177';
$rs = file_get_contents($url);
// echo $rs;exit;
preg_match_all('/(\/pic\/series-s15306\/289-\d+\.html)/', $rs, $urlArr);

$avalie = array_unique($urlArr[0]);
$count = array();
foreach ($avalie as $key => $ul) {
   $pattern = '/;
   preg_match_all($pattern, file_get_contents('http://car.autohome.com.cn'.$ul), $imgSrc);
   $count = array_merge($count, $imgSrc[1]);
}


foreach($count as $k=>$v) {
  $data[$k] = file_get_contents($v);
}

foreach($data as $k=>$v) {
  file_put_contents('./pic2/'.time().'_'.rand(1, 10000).'.jpg', $v);
}

$runtime->stop();  
echo "页面执行时间: ".$runtime->spent()." 毫秒";

Copy after login

　 It turns out that this method is fine with fewer pictures, but it is quite laggy if there are too many pictures. . It is also difficult to run local tests, let alone go online by then. After Baidu, I used the curl method to download images. After testing, it did improve, but it still felt a bit slow. It would be great if PHP had multiple threads. . .

After some tossing and looking for information, I found that the curl library of PHP can actually simulate multi-threading, that is, using the curl_multi_* series of functions. After rewriting, the code became like this:

php
header('Content-type:text/html;charset=utf-8');
set_time_limit(0);

class runtime  
{  
    var $StartTime = 0;  
    var $StopTime = 0;  
   
    function get_microtime()  
    {  
        list($usec, $sec) = explode(' ', microtime());  
        return ((float)$usec + (float)$sec);  
    }  
   
    function start()  
    {  
        $this->StartTime = $this->get_microtime();  
    }  
   
    function stop()  
    {  
        $this->StopTime = $this->get_microtime();  
    }  
   
    function spent()  
    {  
        return round(($this->StopTime - $this->StartTime) * 1000, 1);  
    }  
   
}  

$runtime= new runtime();  
$runtime->start();  


$url = 'http://car.autohome.com.cn/pic/series-s15306/289.html#pvareaid=102177';
$rs = file_get_contents($url);
preg_match_all('/(\/pic\/series-s15306\/289-\d+\.html)/', $rs, $urlArr);

$avalie = array_unique($urlArr[0]);
$count = array();
foreach ($avalie as $key => $ul) {
   $pattern = '/;
   preg_match_all($pattern, file_get_contents('http://car.autohome.com.cn'.$ul), $imgSrc);
   $count = array_merge($count, $imgSrc[1]);
}

$handle = curl_multi_init();

foreach($count as $k => $v) {
  $curl[$k] = curl_init($v);
  curl_setopt($curl[$k], CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($curl[$k], CURLOPT_HEADER, 0);
  curl_setopt($curl[$k], CURLOPT_TIMEOUT, 30);
  curl_multi_add_handle ($handle, $curl[$k]);
}

$active = null;

do {
    $mrc = curl_multi_exec($handle, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);

while ($active && $mrc == CURLM_OK) {
    // 这句在php5.3以后的版本很关键，因为没有这句，可能curl_multi_select可能会永远返回-1，这样就永远死在循环里了
    while (curl_multi_exec($handle, $active) === CURLM_CALL_MULTI_PERFORM);

    if (curl_multi_select($handle) != -1) {
        do {
            $mrc = curl_multi_exec($handle, $active);
        } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    }
}

foreach ($curl as $k => $v) {
    if (curl_error($curl[$k]) == "") {
        $data[$k] = curl_multi_getcontent($curl[$k]);
    }
    curl_multi_remove_handle($handle, $curl[$k]);
    curl_close($curl[$k]);
}

foreach($data as $k=>$v) {
    $file = time().'_'.rand(1000, 9999).'.jpg';
    file_put_contents('./pic3/'.$file, $v); 
}

curl_multi_close($handle);

$runtime->stop();  
echo "页面执行时间: ".$runtime->spent()." 毫秒";

Copy after login

Okay, multi-threaded collection is really refreshing. After a series of tests and comparisons, 5 tests showed that curl multi-threading was faster than file_get_content 4 times, and the time was 3 to 5 times that of file_get_content. To sum up, , try to use this method in future collections to improve efficiency.

The above introduces the performance comparison of using the file_get_content series of functions and using the curl series of functions to collect images, including the relevant aspects. I hope it will be helpful to friends who are interested in PHP tutorials.