How to crawl and analyze web pages with PHP-PHP Tutorial-php.cn

Home

Backend Development

PHP Tutorial

How to crawl and analyze web pages with PHP

不言

Jun 07, 2018 pm 03:45 PM

php

This article mainly introduces the method of PHP crawling and analyzing web pages. It analyzes the principles of PHP web crawling and analysis techniques in detail in the form of examples. Friends in need can refer to the following

The examples of this article are explained Learn how to crawl and analyze web pages with PHP. Sharing it with everyone for your reference, the details are as follows:

Crawling and analyzing a file is very simple. This tutorial will take you step by step through an example to implement it. let's start!

First, we must decide which URL address we will crawl. This can be set in a script or passed via $QUERY_STRING. For simplicity, let's set the variable directly in the script.

<?php
$url = &#39;http://www.php.net&#39;;
?>

The second step is to grab the specified file and store it in an array through the file() function.

<?php
$url = &#39;http://www.php.net&#39;;
$lines_array = file($url);
?>

Okay, now there are files in the array. However, the text we want to analyze may not all be in one line. To resolve this file, we can simply convert the array $lines_array into a string. We can use the implode(x,y) function to achieve this. If you want to use explode later (array of string variables), it may be better to set x to "|" or "!" or other similar delimiter. But for our purposes, it's best to set x to a space. y is another necessary parameter because it is the array you want to process with implode().

<?php
$url = &#39;http://www.php.net&#39;;
$lines_array = file($url);
$lines_string = implode(&#39;&#39;, $lines_array);
?>

Now that the crawling work is done, it’s time to analyze. For the purposes of this example, we want to get everything between

and . In order to parse out the string, we also need something called a regular expression.

<?php
$url = &#39;http://www.php.net&#39;;
$lines_array = file($url);
$lines_string = implode(&#39;&#39;, $lines_array);
eregi("<head>(.*)</head>", $lines_string, $head);
?>

Let’s take a look at the code. As you can see, the eregi() function is executed in the following format:

eregi("<head>(.*)</head>", $lines_string, $head);

"(.*)" means everything and can be interpreted as, "analysis Everything between

and ". $lines_string is the string we are analyzing, and $head is the array where the analyzed results are stored.

Finally, we can enter the data. Since there is only one instance between

and , we can safely assume that there is only one element in the array, and it is the one we want. Let's print it out.

This is all the code.

<?php
//获取所有内容url保存到文件
function get_index ( $save_file , $prefix = "index_" ){
   $count = 68 ;
   $i = 1 ;
  if ( file_exists ( $save_file )) @ unlink ( $save_file );
   $fp = fopen ( $save_file , "a+" ) or die( "Open " . $save_file . " failed" );
  while( $i < $count ){
     $url = $prefix . $i . ".htm" ;
    echo "Get " . $url . "..." ;
     $url_str = get_content_url ( get_url ( $url ));
    echo " OK/n" ;
     fwrite ( $fp , $url_str );
    ++ $i ;
  }
   fclose ( $fp );
}
//获取目标多媒体对象
function get_object ( $url_file , $save_file , $split = "|--:**:--|" ){
  if (! file_exists ( $url_file )) die( $url_file . " not exist" );
   $file_arr = file ( $url_file );
  if (! is_array ( $file_arr ) || empty( $file_arr )) die( $url_file . " not content" );
   $url_arr = array_unique ( $file_arr );
  if ( file_exists ( $save_file )) @ unlink ( $save_file );
   $fp = fopen ( $save_file , "a+" ) or die( "Open save file " . $save_file . " failed" );
  foreach( $url_arr as $url ){
    if (empty( $url )) continue;
    echo "Get " . $url . "..." ;
     $html_str = get_url ( $url );
    echo $html_str ;
    echo $url ;
    exit;
     $obj_str = get_content_object ( $html_str );
    echo " OK/n" ;
     fwrite ( $fp , $obj_str );
  }
   fclose ( $fp );
}
//遍历目录获取文件内容
function get_dir ( $save_file , $dir ){
   $dp = opendir ( $dir );
  if ( file_exists ( $save_file )) @ unlink ( $save_file );
   $fp = fopen ( $save_file , "a+" ) or die( "Open save file " . $save_file . " failed" );
  while(( $file = readdir ( $dp )) != false ){
    if ( $file != "." && $file != ".." ){
      echo "Read file " . $file . "..." ;
       $file_content = file_get_contents ( $dir . $file );
       $obj_str = get_content_object ( $file_content );
      echo " OK/n" ;
       fwrite ( $fp , $obj_str );
    }
  }
   fclose ( $fp );
}
//获取指定url内容
function get_url ( $url ){
   $reg = &#39;/^http:////[^//].+$/&#39; ;
  if (! preg_match ( $reg , $url )) die( $url . " invalid" );
   $fp = fopen ( $url , "r" ) or die( "Open url: " . $url . " failed." );
  while( $fc = fread ( $fp , 8192 )){
     $content .= $fc ;
  }
   fclose ( $fp );
  if (empty( $content )){
    die( "Get url: " . $url . " content failed." );
  }
  return $content ;
}
//使用socket获取指定网页
function get_content_by_socket ( $url , $host ){
   $fp = fsockopen ( $host , 80 ) or die( "Open " . $url . " failed" );
   $header = "GET /" . $url . " HTTP/1.1/r/n" ;
   $header .= "Accept: */*/r/n" ;
   $header .= "Accept-Language: zh-cn/r/n" ;
   $header .= "Accept-Encoding: gzip, deflate/r/n" ;
   $header .= "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Maxthon; InfoPath.1; .NET CLR 2.0.50727)/r/n" ;
   $header .= "Host: " . $host . "/r/n" ;
   $header .= "Connection: Keep-Alive/r/n" ;
   //$header .= "Cookie: cnzz02=2; rtime=1; ltime=1148456424859; cnzz_eid=56601755-/r/n/r/n";
   $header .= "Connection: Close/r/n/r/n" ;
   fwrite ( $fp , $header );
   while (! feof ( $fp )) {
     $contents .= fgets ( $fp , 8192 );
   }
   fclose ( $fp );
   return $contents ;
}
//获取指定内容里的url
function get_content_url ( $host_url , $file_contents ){
   //$reg = &#39;/^(#|<a href="http://lib.csdn.net/base/18" class=&#39;replace_word&#39; title="JavaScript知识库" target=&#39;_blank&#39; style=&#39;color:#df3434; font-weight:bold;&#39;>JavaScript</a>.*?|ftp:////.+|http:////.+|.*?href.*?|play.*?|index.*?|.*?asp)+$/i&#39;;
   //$reg = &#39;/^(down.*?/.html|/d+_/d+/.htm.*?)$/i&#39;;
   $rex = "/([hH][rR][eE][Ff])/s*=/s*[&#39;/"]*([^>&#39;/"/s]+)[/"&#39;>]*/s*/i" ;
   $reg = &#39;/^(down.*?/.html)$/i&#39; ;
   preg_match_all ( $rex , $file_contents , $r );
   $result = "" ; //array();
   foreach( $r as $c ){
    if ( is_array ( $c )){
      foreach( $c as $d ){
        if ( preg_match ( $reg , $d )){ $result .= $host_url . $d . "/n" ; }
      }
    }
  }
  return $result ;
}
//获取指定内容中的多媒体文件
function get_content_object ( $str , $split = "|--:**:--|" ){
   $regx = "/href/s*=/s*[&#39;/"]*([^>&#39;/"/s]+)[/"&#39;>]*/s*(.*?<//b>)/i" ;
   preg_match_all ( $regx , $str , $result );
  if ( count ( $result ) == 3 ){
     $result [ 2 ] = str_replace ( "多媒体： " , "" , $result [ 2 ]);
     $result [ 2 ] = str_replace ( " " , "" , $result [ 2 ]);
     $result = $result [ 1 ][ 0 ] . $split . $result [ 2 ][ 0 ] . "/n" ;
  }
  return $result ;
}
?>

The above is the entire content of this article. I hope it will be helpful to everyone’s study. For more related content, please pay attention to the PHP Chinese website !

Related recommendations:

thinkphp implements paging display function

The above is the detailed content of How to crawl and analyze web pages with PHP. For more information, please follow other related articles on the PHP Chinese website!

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

PHP: An Introduction to the Server-Side Scripting LanguageApr 16, 2025 am 12:18 AM

PHP is a server-side scripting language used for dynamic web development and server-side applications. 1.PHP is an interpreted language that does not require compilation and is suitable for rapid development. 2. PHP code is embedded in HTML, making it easy to develop web pages. 3. PHP processes server-side logic, generates HTML output, and supports user interaction and data processing. 4. PHP can interact with the database, process form submission, and execute server-side tasks.

PHP and the Web: Exploring its Long-Term ImpactApr 16, 2025 am 12:17 AM

PHP has shaped the network over the past few decades and will continue to play an important role in web development. 1) PHP originated in 1994 and has become the first choice for developers due to its ease of use and seamless integration with MySQL. 2) Its core functions include generating dynamic content and integrating with the database, allowing the website to be updated in real time and displayed in personalized manner. 3) The wide application and ecosystem of PHP have driven its long-term impact, but it also faces version updates and security challenges. 4) Performance improvements in recent years, such as the release of PHP7, enable it to compete with modern languages. 5) In the future, PHP needs to deal with new challenges such as containerization and microservices, but its flexibility and active community make it adaptable.

Why Use PHP? Advantages and Benefits ExplainedApr 16, 2025 am 12:16 AM

The core benefits of PHP include ease of learning, strong web development support, rich libraries and frameworks, high performance and scalability, cross-platform compatibility, and cost-effectiveness. 1) Easy to learn and use, suitable for beginners; 2) Good integration with web servers and supports multiple databases; 3) Have powerful frameworks such as Laravel; 4) High performance can be achieved through optimization; 5) Support multiple operating systems; 6) Open source to reduce development costs.

Debunking the Myths: Is PHP Really a Dead Language?Apr 16, 2025 am 12:15 AM

PHP is not dead. 1) The PHP community actively solves performance and security issues, and PHP7.x improves performance. 2) PHP is suitable for modern web development and is widely used in large websites. 3) PHP is easy to learn and the server performs well, but the type system is not as strict as static languages. 4) PHP is still important in the fields of content management and e-commerce, and the ecosystem continues to evolve. 5) Optimize performance through OPcache and APC, and use OOP and design patterns to improve code quality.

The PHP vs. Python Debate: Which is Better?Apr 16, 2025 am 12:03 AM

PHP and Python have their own advantages and disadvantages, and the choice depends on the project requirements. 1) PHP is suitable for web development, easy to learn, rich community resources, but the syntax is not modern enough, and performance and security need to be paid attention to. 2) Python is suitable for data science and machine learning, with concise syntax and easy to learn, but there are bottlenecks in execution speed and memory management.

PHP's Purpose: Building Dynamic WebsitesApr 15, 2025 am 12:18 AM

PHP is used to build dynamic websites, and its core functions include: 1. Generate dynamic content and generate web pages in real time by connecting with the database; 2. Process user interaction and form submissions, verify inputs and respond to operations; 3. Manage sessions and user authentication to provide a personalized experience; 4. Optimize performance and follow best practices to improve website efficiency and security.

PHP: Handling Databases and Server-Side LogicApr 15, 2025 am 12:15 AM

PHP uses MySQLi and PDO extensions to interact in database operations and server-side logic processing, and processes server-side logic through functions such as session management. 1) Use MySQLi or PDO to connect to the database and execute SQL queries. 2) Handle HTTP requests and user status through session management and other functions. 3) Use transactions to ensure the atomicity of database operations. 4) Prevent SQL injection, use exception handling and closing connections for debugging. 5) Optimize performance through indexing and cache, write highly readable code and perform error handling.

How do you prevent SQL Injection in PHP? (Prepared statements, PDO)Apr 15, 2025 am 12:15 AM

Using preprocessing statements and PDO in PHP can effectively prevent SQL injection attacks. 1) Use PDO to connect to the database and set the error mode. 2) Create preprocessing statements through the prepare method and pass data using placeholders and execute methods. 3) Process query results and ensure the security and performance of the code.

See all articles

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)

4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

R.E.P.O. Best Graphic Settings

4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Assassin's Creed Shadows: Seashell Riddle Solution

2 weeks agoByDDD

R.E.P.O. How to Fix Audio if You Can't Hear Anyone

4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

R.E.P.O. Chat Commands and How to Use Them

4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

VSCode Windows 64-bit Download

A free and powerful IDE editor launched by Microsoft

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software