How to use PHP to implement regular crawling of URLs in pages

Grab all the links in the page from the page. Of course, using PHP regular expressions is the most convenient way. To write a regular expression, you must first summarize the pattern. So how many forms will the links on the page have? Let’s take a look below.

Preface

A link is a hyperlink, which is a link from one element (text, picture, video, etc.) to another element (text, picture, video, etc.) . There are generally three types of links in web pages. One is an absolute URL hyperlink, which is the complete path to a page; the other is a relative URL hyperlink, which generally links to other pages on the same website; and the other is a hyperlink within a page. Hyperlink, which generally links to other locations within the same page.

Once you understand the types of links, you will know that to grab links, the main ones are absolute URL hyperlinks and relative URL hyperlinks. To write correct regular expressions, we must understand the pattern of the object we are looking for.

Let’s start with an absolute link, also called a URL (Uniform Resource Locator), which identifies a unique resource on the Internet. The structure of the URL contains three parts: protocol, server name, path and file name.

The protocol is the identifier that tells the browser how to handle the file to be opened. The most common one is the http protocol. This article only considers the HTTP protocol. As for other https, ftp, mailto, telnet protocols, etc., you can also add them as needed.

The server name is how to tell the browser how to reach the server. It is usually a domain name or IP address, and sometimes includes a port number (default is 80). In the FTP protocol, user names and passwords can also be included, which will not be considered in this article.

Path and file name, usually separated by /, indicate the path to the file and the name of the file itself. If there is no specific file name, the default file in this folder is accessed (can be set on the server side).

So now it is clear that the typical form of absolute links to be crawled can be summarized as

<span style="color: #000000">http://www.xxx.com/xxx/yyy/zzz .html</span>

The character range that can be used in each part has clear specifications. For details, please refer to RFC1738. Then the regular expression can be written.

/(http|https):\/\/([\w\d\-_]+[\.\w\d\-_]+)[:\d+]?([\/]?[\w\/\.]+)/i

The explanation is as follows:

(http|https)The first bracket matches the protocol part.

([\w\d\-_] [\.\w\d\-_] )The second bracket matches the domain name part.

([\/]?[\w\/\.] )The third bracket matches the relative path.

At this time of writing, basically most of the URLs can be matched, but URLs with parameters cannot be crawled, which may cause the page to report an error when accessing again. What parameters are required in the RFC1738 specification? to split, followed by parameters, but modern RIA applications may use other strange forms for splitting.

Modify it slightly so that you can search out the query parameter part. This still does not cover all situations, such as situations where the URL contains Chinese characters, spaces, and other special characters, but it basically meets my needs, so I will not go further.

/(http|ftp|https):\/\/([\w\d\-_]+[\.\w\d\-_]+)[:\d+]?([\/]?[\w\/\.\?=&;%@#\+,]+)/i

The advantage of using parentheses is that when processing the results, you can easily obtain the protocol, domain name, and relative path, which facilitates subsequent processing.

For example, use preg_match_all() When matching, the result array index 0 is all results, 1 is protocol, 2 is domain name, and 3 is relative path .

Summary: The above is the entire content of this article, I hope it will be helpful to everyone's study.

Related recommendations:

How to implement multi-field fuzzy matching query in PHP

phpmailer How to implement binding mailbox

How to implement custom function in PHP to obtain the first letter of Chinese characters

The above is the detailed content of How to use PHP to implement regular crawling of URLs in pages. For more information, please follow other related articles on the PHP Chinese website!