Learning and application of CURL (with multi-thread implementation)

curl installation:

Installation under windows
: Modify the settings of the php.ini file and find php_curl.dll
//Cancel the comment below extension= php_curl.dll
Installed under Linux:

Copy code The code is as follows:

# wget http://curl.haxx.se/download/curl-7.17.1.tar.gz
# tar zxvf curl-7.17.1.tar.gz //Unzip
#cd curl-7.17.1
# ./configure –prefix=/usr/local/curl
# make
# make install

This is the method to install before installing php.
***************************phpinf Check whether loading is successful!
Use curl's POST data Fetion interface
Use curl to write the Fetion interface. There are many on the Internet. Here is just a test

Copy the code The code is as follows:

$username = 13800138000;
$password = 123456;
$sendto = 13912345678;
$message = "Test one and give it a try!";
 $curlPost = 'username='.urlencode($username).'&password='.urlencode($password).'&sendto='.urlencode($sendto).'&message='.urlencode($message).''; 
$ch = curl_init();//Initialize curl
curl_setopt($ch,CURLOPT_URL,'http://sms.api.bz/fetion.php');//Catch the specified web page
 curl_setopt($ch, CURLOPT_HEADER, 0);//Set header
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);//Require the result to be a string and output it to the screen
curl_setopt($ch, CURLOPT_POST, 1 );//Post submission method
curl_setopt($ch, CURLOPT_POSTFIELDS, $curlPost);
$data = curl_exec($ch);//Run curl
curl_close($ch);
print_r ($data);//Output result

The returned result is: the text message has been submitted to the sending queue!
The address of Fetion interface is http://sms.api.bz/
Fetion interface mode:
http://sms.api.bz/fetion.php?username=Your mobile Fetion login phone Number
&password=Your mobile Fetion login password
&sendto=Fetion friend’s mobile phone number to receive the text message
&message=SMS content
Format: http://sms.api.bz/fetion.php? username=13800138000&password=123456&sendto=13912345678&message=SMS content
Be careful to keep the utf-8 format, I made a mistake on this point

To summarize the curl method:

Initialize curl

Use curl_setopt to set the target url, and other options. For details on these options, please refer to: http://cn2.php.net/manual/zh/ref.curl.php

curl_exec, execute curl

After execution, close curl
The last step is to output
one of the most important curl functions: curl_getinfo
curl_getinfo ( resource $ch [, int $opt = 0 ] )

Copy code The code is as follows:

/*curl instance
*/
$curl = curl_init();
// Set the URL you need to crawl
curl_setopt($curl, CURLOPT_URL, 'http://www.baidu.com');
// Set header
curl_setopt($curl, CURLOPT_HEADER, 0);
//Set cURL parameters to ask whether the results are saved in a string or output to the screen.
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
// Run cURL and request the web page
$data = curl_exec($curl);
if($data === false){
echo curl_error($curl);exit;
}
$info = curl_getinfo($curl);
// Close URL request
curl_close($curl);

// Display the obtained data
var_dump($info);
var_dump($data);

can return:

URLINFO_EFFECTIVE_URL - the last valid URL address
CURLINFO_HTTP_CODE - the last received HTTP code
CURLINFO_FILETIME - the time the document was retrieved remotely, if it cannot be retrieved, The return value is "-1"
CURLINFO_TOTAL_TIME - the time spent on the last transmission
CURLINFO_NAMELOOKUP_TIME - the time spent on name resolution
CURLINFO_CONNECT_TIME - the time spent on establishing the connection
CURLINFO_PRETRANSFER_TIME - the time spent on establishing the connection Time taken to prepare for transfer
CURLINFO_STARTTRANSFER_TIME – Time taken from establishment of connection to start of transfer
CURLINFO_REDIRECT_TIME – Time taken to redirect before transaction transfer starts
CURLINFO_SIZE_UPLOAD – Total amount of data uploaded< + CURLINFO_REQUEST_SIZE – Size of the request in question in the HTTP request
CURLINFO_SSL_VERIFYRESULT – The result of the SSL certificate verification request returned by setting CURLOPT_SSL_VERIFYPEER
CURLINFO_CONTENT_LENGTH_DOWNLOAD – The download content length read from the Content-Length: field
CURLINFO_CONTENT_LENGTH_UPLOAD – Upload Description of content size
CURLINFO_CONTENT_TYPE – Content-Type: value of downloaded content, NULL means the server did not send a valid Content-Type: header

Use curl to implement multi-threading

curl is generally used To capture web pages, the second is to get or post data, and the third application is to implement multi-threaded tasks of PHP
Next, we will implement multi-threaded

copy Code

The code is as follows:

/*
curl multi-threaded crawling
*/
/**
* curl multi-threading
* * @param array $array parallel URL
* @param int $timeout timeout
* @return array
*/
function Curl_http($array,$timeout ){
$res = array();
$mh = curl_multi_init();//Create multiple curl handles
$startime = getmicrotime();
foreach($array as $k =>$url){
$conn[$k]=curl_init($url);

curl_setopt($conn[$k], CURLOPT_TIMEOUT, $timeout);//Set the timeout
curl_setopt($conn[$k], CURLOPT_USERAGENT, 'Mozilla/5.0 (compatible; MSIE 5.01; Windows NT 5.0)');
curl_setopt($conn[$k], CURLOPT_MAXREDIRS, 7);//HTTp Orientation level
curl_setopt($conn[$k], CURLOPT_HEADER, 0);//No header here, block efficiency
curl_setopt($conn[$k], CURLOPT_FOLLOWLOCATION, 1); // 302 redirect
curl_setopt($conn[$k],CURLOPT_RETURNTRANSFER,1);
curl_multi_add_handle ($mh,$conn[$k]);
}
//Prevent endless loops from consuming the CPU. This section is According to the writing method on the Internet
do {
$mrc = curl_multi_exec($mh,$active);//When there is no data, active=true
} while ($mrc == CURLM_CALL_MULTI_PERFORM);//When there is no data When accepting data
while ($active and $mrc == CURLM_OK) {//When there is no data or the request is paused, active=true
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}

foreach ($array as $k => $url) {
curl_error($conn[$k]);
$res[$k]=curl_multi_getcontent($conn[$k]);//Get the return information
$header[$k]=curl_getinfo($conn[$k]);//Return header information
curl_close($conn[$k]);//Close handle
curl_multi_remove_handle($mh, $conn [$k]); //Release resources
}

curl_multi_close($mh);
$endtime = getmicrotime();
$diff_time = $endtime - $startime;

return array('diff_time'=>$diff_time,
'return'=>$res,
'header'=>$header
);

}
//Calculate the current time
function getmicrotime() {
list($usec, $sec) = explode(" ",microtime());
return ((float)$usec + (float)$sec);
}

//Test it, curl three URLs
$array = array(
"http://www.weibo.com/",
"http://www.renren.com/",
"http://www.qq.com/"
);
$data = Curl_http($array,'10' ); // Call
var_dump($data); // Output

?>

Explanation about do while:

Because $active has to wait until all url data is received before it becomes false, so the return value of curl_multi_exec is used here to determine whether there is still data,
When there is data, curl_multi_exec will be called continuously. If there is no data for the time being, it will enter the select phase. Once new data comes, it can be awakened to continue execution.
The advantage here is that the unnecessary consumption of CPU is gone. More detailed description: http://hi.baidu.com/%D4%C2%D2%B9%C4%FD%ED%F8/blog/item/9dfcf4fbe6b84374024f563d.html

The steps to write this multi-thread:
Step 1: Call curl_multi_init
Step 2: Call curl_multi_add_handle in a loop
It should be noted in this step that the second parameter of curl_multi_add_handle comes from curl_init subhandle.
Step 3: Continue to call curl_multi_exec
Step 4: Call curl_multi_getcontent in a loop to obtain the results as needed
Step 5: Call curl_multi_remove_handle, and call curl_close for each word handle
Step 6: Call curl_multi_close
Multi-threaded test renderings:

Learning and application of CURL (with multi-thread implementation)_PHP tutorial

Summary: 36 http requests, judging from the chronological order of execution, the IPs of the three websites intersect, indicating that they are concurrent at the same time!
——————————————————————————
curl under linux command
Several common usage methods:
Download Function:
Direct download is equivalent to wget
curl -o 1.jpg http://cgi2.tky.3web.ne.jp/~zzh/screen1.JPG
Batch download screen1.JPG–screen10. JPG
curl -O http://cgi2.tky.3web.ne.jp/~zzh/screen[1-10].JPG
Breakpoint download
curl -c -O http:// cgi2.tky.3wb.ne.jp/~zzh/screen1.JPG

Reverse proxy function
curl -x 123.45.67.89:1080 -o page.html http://www.yahoo.com
Show header file
curl -I www.sina .com