Practical crawler practice: Use PHP to crawl Ctrip travel information

With the continuous development of tourism, tourism information has become very rich. In order to facilitate everyone to obtain more comprehensive and accurate travel information, we can use crawlers to crawl data on travel websites, analyze and process it. This article will introduce how to use PHP to crawl Ctrip travel information.

1. Basics of crawlers

A crawler is an automated program that simulates users visiting a website and obtains data on the website. The crawler is generally divided into the following steps:

1. Initiate a request: The crawler program will initiate an HTTP request to the target website to obtain the HTML code of the target website.
2. Parse HTML: Use an appropriate parser to parse the HTML code and obtain the required data. Common parsers include regular expressions, XPath and DOM parsers.
3. Storing data: Store the obtained data in a file or database for subsequent analysis and processing.
4. Target website analysis

Before writing a crawler, you first need to analyze the structure and data of the target website. The target website of this crawler is the Ctrip travel website. We need to analyze the URL structure and HTML code of the website to determine how it is requested and how it is parsed.

2.1 URL structure analysis

The URL structure of Ctrip’s travel website is relatively simple. Each travel product has a unique ID number. Take browsing Beijing tourism as an example. The URL is: https://you.ctrip.com/sight/beijing1.html. Among them, beijing1 represents the ID number of Beijing Tourism.

2.2 HTML code analysis

Use the developer tools of the Chrome browser to view the HTML code of the webpage. You can find that the travel information on the page is in a div with the class "list_mod2". This div contains detailed information about each travel product, including product name, price, reviews, etc. In the crawler program, we need to use PHP's DOM parser to parse this information.

3. Crawler program writing

Now that we have understood the URL structure and HTML code of the target website, we can write a crawler program to obtain travel information.

3.1 Initiate an HTTP request

First, we need to use PHP's cURL library to initiate an HTTP request to the target website and obtain the HTML code of the target website. In the code, we use a loop, starting from ID=1, to obtain the detailed information of each travel product in sequence.

//爬取ID号从1到n的旅游产品的详细信息 for($i=1;$i<=$n;$i++){ $url = "https://you.ctrip.com/sight/beijing$i.html"; //目标网站URL $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $html = curl_exec($ch); curl_close($ch); }

3.2 Parse HTML code

Next, we use PHP’s DOM parser to parse the HTML code of the target website and obtain the travel product information we need. In the code, we use XPath expressions to select the div with class "list_mod2" and obtain the product name, price, evaluation and other information.

$doc = new DOMDocument(); $doc->loadHTML($html); $xpath = new DOMXPath($doc); $list_mod2 = $xpath->query('//*[@class="list_mod2"]'); //选择class为"list_mod2"的div foreach($list_mod2 as $mod){ $modImg = $xpath->query('a/img/@src', $mod)->item(0)->nodeValue; //获取产品图片URL $modTitle = $xpath->query('div/h2/a', $mod)->item(0)->nodeValue; //获取产品名称 $modContent = $xpath->query('div/div/div/p/@title', $mod)->item(0)->nodeValue; //获取产品简介 $modPrice = $xpath->query('div/div/span/em', $mod)->item(0)->nodeValue; //获取产品价格 $modComment = $xpath->query('div/div/div/div/span[1]', $mod)->item(0)->nodeValue; //获取产品评价分数 $modCommentNum = $xpath->query('div/div/div/div/span[2]', $mod)->item(0)->nodeValue; //获取产品评价人数 }

3.3 Storing data

Finally, we store the obtained tourism product information in the database. In the code, we use PHP's PDO extension to connect to the database and insert the obtained product information into the database table named "product_info".

//连接数据库 $dsn = 'mysql:host=localhost;port=3306;dbname=test;charset=utf8'; $username = 'root'; $password = '123456'; $options = array(PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION); $dbh = new PDO($dsn, $username, $password, $options); //将获取到的产品信息插入到数据库 $stmt = $dbh->prepare("INSERT INTO product_info(img_url, title, content, price, comment_score, comment_num) VALUES(?, ?, ?, ?, ?, ?)"); $stmt->bindParam(1, $modImg); $stmt->bindParam(2, $modTitle); $stmt->bindParam(3, $modContent); $stmt->bindParam(4, $modPrice); $stmt->bindParam(5, $modComment); $stmt->bindParam(6, $modCommentNum); $stmt->execute();

4. Summary

Using PHP to write a crawler program, you can easily grab the travel product information of Ctrip travel website and store it in the database. In actual development, we can also use multi-threading, IP proxy and other technologies to improve the efficiency and stability of the crawler program. However, it should be noted that when using crawler programs, the copyright and privacy policies of the target website should be respected to avoid infringement and abuse.

The above is the detailed content of Practical crawler practice: Use PHP to crawl Ctrip travel information. For more information, please follow other related articles on the PHP Chinese website!