Can php be used as a crawler?

phpspider An excellent PHP development spider crawler

##To write a PHP web crawler, you need to have the following skills:

The crawler is written in PHP (recommended learning:

PHP video tutorial)

Extracting data from web pages requires XPath (XPath selector tutorial)

Of course we can also use CSS selectors (CSS selector tutorial)

Regular expressions (regular expression tutorial) are used in many cases

Chrome’s developer tools are artifacts , many AJAX requests need to be analyzed using it

Note:This framework can only be run under the command line, command line, command line, command line, important things must be said three times^_ ^

The demo written in this article is to crawl the military education website

<?php
require_once __DIR__ . &#39;/../autoloader.php&#39;;
use phpspider\core\phpspider;

/* Do NOT delete this comment */
/* 不要删除这段注释 */

$configs = array(
    &#39;name&#39; => &#39;军事&#39;, // 给你的爬虫起一个名字
    &#39;log_show&#39; => false, // 是否显示日志
    &#39;tasknum&#39; => 1, // 开启多少个进程爬取
    // 数据库配置
    &#39;db_config&#39; => array(
        &#39;host&#39;  => &#39;127.0.0.1&#39;,
        &#39;port&#39;  => 3306,
        &#39;user&#39;  => &#39;root&#39;,
        &#39;pass&#39;  => &#39;root&#39;,
        &#39;name&#39;  => &#39;collection&#39;,
    ),
    // 数据库表，表需要已存在，collection库，test表
    &#39;export&#39; => array(
        &#39;type&#39; => &#39;db&#39;,
        &#39;table&#39; => &#39;test&#39;,
    ),
    // 爬取的域名列表
    &#39;domains&#39; => array(
        &#39;war.163.com&#39;
    ),
    // 抓取的起点
    &#39;scan_urls&#39; => array(
        &#39;http://war.163.com&#39;
    ),
    // 列表页实例，你要爬取的列表，也就是分页
    &#39;list_url_regexes&#39; => array(
        "http://war.163.com"
    ),
    // 内容页实例，文章的内容页
    // \d+ 指的是变量，就是可变的参数
    &#39;content_url_regexes&#39; => array(
        "http://war.163.com/photoview/4T8E0001/\d+",
    ),
    // 失败重新爬取次数
    &#39;max_try&#39; => 5,
    // 爬取规则配置
    &#39;fields&#39; => array(
        array(
            &#39;name&#39; => "title", // 数据库字段名
            &#39;selector&#39; => "//div[@class=&#39;headline&#39;]/h1", // 规则，表示：headline类里的h1标签
            &#39;required&#39; => true, // 如果为空，整条数据丢弃
        ),
        array(
            &#39;name&#39; => "content",
            &#39;selector&#39; => "//div[@class=&#39;overview&#39;]/p",
            &#39;required&#39; => true,
        ),
        array(
            &#39;name&#39; => "img",
            &#39;selector&#39; => "//img[@class=&#39;firstPreload&#39;]",
            &#39;required&#39; => true,
        ),
    ),
);

$spider = new phpspider($configs);
$spider->start();

The above is the detailed content of Can php be used as a crawler?. For more information, please follow other related articles on the PHP Chinese website!