A brief analysis of how to use PHP to crawl data asynchronously-PHP Tutorial-php.cn

Speaking of crawlers, many people will think of python crawlers, because it does have great advantages. But in fact, PHP can also be used to crawl data asynchronously. Let me introduce to you how to use PHP to crawl data asynchronously.

A brief analysis of how to use PHP to crawl data asynchronously

What is a web crawler?

A web crawler is a program that automatically extracts web pages. It downloads web pages from the World Wide Web for search engines and is an important component of search engines. The traditional crawler starts from the URL of one or several initial web pages and obtains the URL on the initial web page. During the process of crawling the web page, it continuously extracts new URLs from the current page and puts them into the queue until certain stopping conditions of the system are met.

What is the use of crawlers?

As a general search engine web page collector. (google, baidu)
Build a vertical search engine.
Scientific research: online human behavior, online community evolution, human dynamics research , Empirical research in fields such as econometric sociology, complex networks, data mining, etc. all require large amounts of data, and web crawlers are a powerful tool for collecting relevant data.
Peeping, hacking, spamming...

QueryList Introduction and Features

QueryList is a set of simple, elegant, and scalable PHP collection tools (crawlers) based on phpQuery.

Features:

Have the exact same CSS3 DOM selector as jQuery
Have the exact same DOM operation as jQuery API
has a universal list collection solution
has a powerful HTTP request suite, which can easily implement: simulated login, fake browser, HTTP Proxy and other complex network requests
Have garbled solution
Have powerful content filtering function, you can use jQuey selector to filter content
Have a high degree of modular design and strong scalability
Have an expressive API
Have high-quality documentation
Have rich plug-ins
Have professional Q&A community and communication group

Through plug-ins, you can easily achieve things such as:

Multi-thread collection
Picture localization
Simulate browser behavior, such as: submit Form form
Web crawler

Environment requirements

PHP >= 7.0

If your PHP version is still stuck at PHP5, or you don’t know how to use Composer, you can choose to use QueryList3. QueryList3 supports php5.3 and manual installation. QueryList3 documentation: http://v3.querylist.cc

Installation

Installation via Composer:

composer require jaeger/querylist

Copy after login

Use

Element operation

Collect all image addresses of "Nitu.com"

QueryList::get(&#39;http://www.nipic.com&#39;)->find(&#39;img&#39;)->attrs(&#39;src&#39;);

Copy after login

Collect Baidu search results

$ql = QueryList::get(&#39;http://www.baidu.com/s?wd=QueryList&#39;);

$ql->find(&#39;title&#39;)->text(); // 获取网站标题
$ql->find(&#39;meta[name=keywords]&#39;)->content; // 获取网站头部关键词

$ql->find(&#39;h3>a&#39;)->texts(); //获取搜索结果标题列表
$ql->find(&#39;h3>a&#39;)->attrs(&#39;href&#39;); //获取搜索结果链接列表

$ql->find(&#39;img&#39;)->src; //获取第一张图片的链接地址
$ql->find(&#39;img:eq(1)&#39;)->src; //获取第二张图片的链接地址
$ql->find(&#39;img&#39;)->eq(2)->src; //获取第三张图片的链接地址
// 遍历所有图片
$ql->find(&#39;img&#39;)->map(function($img){
    echo $img->alt;  //打印图片的alt属性
});

Copy after login

More usage

$ql->find(&#39;#head&#39;)->append(&#39;<div>追加内容</div>&#39;)->find(&#39;div&#39;)->htmls();
$ql->find(&#39;.two&#39;)->children(&#39;img&#39;)->attrs(&#39;alt&#39;); //获取class为two元素下的所有img孩子节点
//遍历class为two元素下的所有孩子节点
$data = $ql->find(&#39;.two&#39;)->children()->map(function ($item){
    //用is判断节点类型
    if($item->is(&#39;a&#39;)){
        return $item->text();
    }elseif($item->is(&#39;img&#39;))
    {
        return $item->alt;
    }
});

$ql->find(&#39;a&#39;)->attr(&#39;href&#39;, &#39;newVal&#39;)->removeClass(&#39;className&#39;)->html(&#39;newHtml&#39;)->...
$ql->find(&#39;div > p&#39;)->add(&#39;div > ul&#39;)->filter(&#39;:has(a)&#39;)->find(&#39;p:first&#39;)->nextAll()->andSelf()->...
$ql->find(&#39;div.old&#39;)->replaceWith( $ql->find(&#39;div.new&#39;)->clone())->appendTo(&#39;.trash&#39;)->prepend(&#39;Deleted&#39;)->...

Copy after login

List collection

Collect the titles and links of Baidu search results list:

$data = QueryList::get(&#39;http://www.baidu.com/s?wd=QueryList&#39;)
    // 设置采集规则
    ->rules([ 
        &#39;title&#39;=>array(&#39;h3&#39;,&#39;text&#39;),
        &#39;link&#39;=>array(&#39;h3>a&#39;,&#39;href&#39;)
    ])
    ->query()->getData();

print_r($data->all());

Copy after login

Collection results:

Array
(
    [0] => Array
        (
            [title] => QueryList|基于phpQuery的无比强大的PHP采集工具
            [link] => http://www.baidu.com/link?url=GU_YbDT2IHk4ns1tjG2I8_vjmH0SCJEAPuuZN
        )
    [1] => Array
        (
            [title] => PHP 用QueryList抓取网页内容 - wb145230 - 博客园
            [link] => http://www.baidu.com/link?url=zn0DXBnrvIF2ibRVW34KcRVFG1_bCdZvqvwIhUqiXaS
        )
    [2] => Array
        (
            [title] => 介绍- QueryList指导文档
            [link] => http://www.baidu.com/link?url=pSypvMovqS4v2sWeQo5fDBJ4EoYhXYi0Lxx
        )
        //...
)

Copy after login

Encoding conversion

// 输出编码:UTF-8,输入编码:GB2312
QueryList::get(&#39;https://top.etao.com&#39;)->encoding(&#39;UTF-8&#39;,&#39;GB2312&#39;)->find(&#39;a&#39;)->texts();

// 输出编码:UTF-8,输入编码:自动识别
QueryList::get(&#39;https://top.etao.com&#39;)->encoding(&#39;UTF-8&#39;)->find(&#39;a&#39;)->texts();

Copy after login

HTTP network operation

//采集新浪微博需要登录才能访问的页面
$ql = QueryList::get(&#39;http://weibo.com&#39;,&#39;param1=testvalue & params2=somevalue&#39;,[
    &#39;headers&#39; => [
        //填写从浏览器获取到的cookie
        &#39;Cookie&#39; => &#39;SINAGLOBAL=546064; wb_cmtLike_2112031=1; wvr=6;....&#39;
    ]
]);
//echo $ql->getHtml();
echo $ql->find(&#39;title&#39;)->text();
//输出: 我的首页 微博-随时随地发现新鲜事

Copy after login

Use Http proxy

$urlParams = [&#39;param1&#39; => &#39;testvalue&#39;,&#39;params2&#39; => &#39;somevalue&#39;];
$opts = [
    // 设置http代理
    &#39;proxy&#39; => &#39;http://222.141.11.17:8118&#39;,
    //设置超时时间，单位：秒
    &#39;timeout&#39; => 30,
     // 伪造http头
    &#39;headers&#39; => [
        &#39;Referer&#39; => &#39;https://querylist.cc/&#39;,
        &#39;User-Agent&#39; => &#39;testing/1.0&#39;,
        &#39;Accept&#39;     => &#39;application/json&#39;,
        &#39;X-Foo&#39;      => [&#39;Bar&#39;, &#39;Baz&#39;],
        &#39;Cookie&#39;    => &#39;abc=111;xxx=222&#39;
    ]
];
$ql->get(&#39;http://httpbin.org/get&#39;,$urlParams,$opts);
// echo $ql->getHtml();

Copy after login

Simulated login

// 用post登录
$ql = QueryList::post(&#39;http://xxxx.com/login&#39;,[
    &#39;username&#39; => &#39;admin&#39;,
    &#39;password&#39; => &#39;123456&#39;
])->get(&#39;http://xxx.com/admin&#39;);
//采集需要登录才能访问的页面
$ql->get(&#39;http://xxx.com/admin/page&#39;);
//echo $ql->getHtml();

Copy after login

Form form operation

Simulate login to GitHub

// 获取QueryList实例
$ql = QueryList::getInstance();
//获取到登录表单
$form = $ql->get(&#39;https://github.com/login&#39;)->find(&#39;form&#39;);

//填写GitHub用户名和密码
$form->find(&#39;input[name=login]&#39;)->val(&#39;your github username or email&#39;);
$form->find(&#39;input[name=password]&#39;)->val(&#39;your github password&#39;);

//序列化表单数据
$fromData = $form->serializeArray();
$postData = [];
foreach ($fromData as $item) {
    $postData[$item[&#39;name&#39;]] = $item[&#39;value&#39;];
}

//提交登录表单
$actionUrl = &#39;https://github.com&#39;.$form->attr(&#39;action&#39;);
$ql->post($actionUrl,$postData);
//判断登录是否成功
// echo $ql->getHtml();
$userName = $ql->find(&#39;.header-nav-current-user>.css-truncate-target&#39;)->text();
if($userName)
{
    echo &#39;登录成功!欢迎你:&#39;.$userName;
}else{
    echo &#39;登录失败!&#39;;
}

Copy after login

Bind function extension

Customize a myHttp method:

$ql = QueryList::getInstance();

//绑定一个myHttp方法到QueryList对象
$ql->bind(&#39;myHttp&#39;,function ($url){
    $html = file_get_contents($url);
    $this->setHtml($html);
    return $this;
});

//然后就可以通过注册的名字来调用
$data = $ql->myHttp(&#39;https://toutiao.io&#39;)->find(&#39;h3 a&#39;)->texts();
print_r($data->all());

Copy after login

Or encapsulate the implementation into a class, and then bind it like this:

$ql->bind(&#39;myHttp&#39;,function ($url){
    return new MyHttp($this,$url);
});

Copy after login

Plug-in usage

Use CURL multi-thread plug-in, multi-thread collection GitHub rankings :

$ql = QueryList::use(CurlMulti::class);
$ql->curlMulti([
    &#39;https://github.com/trending/php&#39;,
    &#39;https://github.com/trending/go&#39;,
    //.....more urls
])
 // 每个任务成功完成调用此回调
 ->success(function (QueryList $ql,CurlMulti $curl,$r){
    echo "Current url:{$r[&#39;info&#39;][&#39;url&#39;]} \r\n";
    $data = $ql->find(&#39;h3 a&#39;)->texts();
    print_r($data->all());
})
 // 每个任务失败回调
->error(function ($errorInfo,CurlMulti $curl){
    echo "Current url:{$errorInfo[&#39;info&#39;][&#39;url&#39;]} \r\n";
    print_r($errorInfo[&#39;error&#39;]);
})
->start([
    // 最大并发数
    &#39;maxThread&#39; => 10,
    // 错误重试次数
    &#39;maxTry&#39; => 3,
]);

Copy after login

For more details, please check GitHub: https://github.com/jae-jae/QueryList

Recommended learning: "PHP Video Tutorial"

The above is the detailed content of A brief analysis of how to use PHP to crawl data asynchronously. For more information, please follow other related articles on the PHP Chinese website!