How to use PHP and phpSpider to crawl and download images?

How to use PHP and phpSpider to crawl and download images?

With the development of the Internet, we have a large number of pictures circulating on the Internet every day. Sometimes we may need to save some pictures locally so that we can view them at any time. Manually downloading one by one may be very tedious and time-consuming. At this time, crawler technology is needed.

This article will introduce how to use PHP language and phpSpider framework to crawl and download images. As a powerful server-side scripting language, PHP is widely used for its advantages of being easy to learn and high development efficiency. phpSpider is a powerful web crawler framework developed based on PHP and is highly scalable and flexible.

First, we need to install the phpSpider framework. Open the terminal and execute the following command:

composer require phpspider/phpspider

After the installation is complete, we can start writing code.

First, create a file named download_img.php, and introduce the entry class of phpSpider into the file:

<?php
require 'vendor/autoload.php';
use phpspidercorephpspider;

Then, we define a class that inherits phpSpider The base class phpspider, and override the handlePage() method for processing page data:

class ImageSpider extends phpspider
{
    public function handlePage($page)
    {
        // 获取图片链接
        $img_urls = $page['rawlinks'];
        
        // 遍历图片链接并将图片下载到本地
        foreach ($img_urls as $img_url) {
            $this->downloadImage($img_url);
        }
    }
    
    private function downloadImage($url)
    {
        // 获取图片文件名
        $file_name = basename($url);
        
        // 构造图片保存路径
        $save_path = './images/' . $file_name;
        
        // 下载图片
        file_put_contents($save_path, file_get_contents($url));
        
        echo '成功下载图片：' . $url . PHP_EOL;
    }
}

Next, we create an index .php file, used to call the ImageSpider class to perform crawling tasks:

<?php
require 'download_img.php';

$spider = new ImageSpider();

// 设置爬虫的配置项
$spider->addUrl('https://www.examplesite.com/');
$spider->notUseCookie();
$spider->start();

In the above code, we first include the previously created download_img.php file and instantiate it The ImageSpider class. Then, we set the configuration items of the crawler, including the initial entry URL to be crawled, not using cookies, etc. Finally, call the start() method to start the crawler task.

The above code will crawl page data starting from the given URL and extract all image links. Then, download these images to the local computer through the downloadImage() method and save them in a folder named images.

Before running this code, we need to create an images folder and ensure that the folder has write permissions.

So far, we have completed how to use PHP and phpSpider to crawl and download images. In this way, we can easily obtain picture resources on the Internet, which is convenient for us to browse and use offline.

To sum up, the process of using PHP and phpSpider to crawl and download images includes four steps: installing the phpSpider framework, creating the main download script file, writing the ImageSpider class to process page data, and setting up the crawler configuration. item and start the crawler task.

I hope this article will help you understand and apply the phpSpider framework, and I wish you a happy use!

The above is the detailed content of How to use PHP and phpSpider to crawl and download images?. For more information, please follow other related articles on the PHP Chinese website!