Use of php data grabbing class Snoopy

PHP Collection Snoopy Detailed Explanation
PHP Collection Tool Snoopy Application Detailed Explanation
Snoopy is a PHP class that is used to simulate the functions of a browser and can obtain web content and send forms. The correct operation of Snoopy requires that your server's PHP version is 4 or above and supports PCRE (Perl Compatible Regular Expressions), which is supported by basic LAMP services.
1. Some features of Snoopy:
　 1. Fetch the content of the web page fetch
2. Fetch the text content of the web page (remove HTML tags) fetchtext
3. Fetch the links of the web page, form fetchlinks fetchform
4. Support proxy host
　5. Support basic username/password verification
　6. Support setting user_agent, referer (source), cookies and header content (header file)
　7. Support browser redirection and control the redirection depth
　8. Can expand the links in the web page into high-quality URLs (default)
　9. Submit data and get the return value
　 10. Support tracking HTML framework
　 11. Support passing cookies when redirecting, it requires PHP4 or above, because It is a PHP class and does not need to be extended. It is the best choice when the server does not support curl.
2. Class method:
fetch($URI)
————–
　This is the method used to grab the content of the web page. The $URI parameter is the URL address of the crawled web page. The fetched results are stored in $this->results. If you are scraping a frame, Snoopy will track each frame and store it in an array, then store it in $this->results.
fetchtext($URI)
——————
This method is similar to fetch(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return the text content in the web page.
fetchform($URI)
——————
This method is similar to fetch(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return the form content (form) in the web page.
fetchlinks($URI)
————-
This method is similar to fetch(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return links in the web page.
By default, relative links will be automatically completed and converted into complete URLs.
submit($URI,$formvars)
——————-
　This method sends a confirmation form to the link address specified by $URL. $formvars is an array that stores form parameters.
submittext($URI,$formvars)
————————–
This method is similar to submit(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return the page after login. text content.
submitlinks($URI)
————-
This method is similar to submit(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return links in the web page.
By default, relative links will be automatically completed and converted into complete URLs.
3. Class attributes: (The default value is in brackets)
$host The connected host
$port The connected port
$proxy_host The proxy host used, if any
$proxy_port The proxy host port used, if any
$agent User agent camouflage (Snoopy v0.1)
$referer source information, if any
$cookies cookies, if any
$rawheaders Other header information, if any
$maxredirs Maximum weight Number of redirects, 0=not allowed (5)
$offsiteok whether or not to allow redirects off-site. (true)
$expandlinks Whether to complete all links to complete addresses (true)
$user authentication user name, if any If so
$pass authentication user name, if any
$accept http acceptance type (image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*)
$error Where to report the error, if any
$response_code The response code returned from the server
$headers The header information returned from the server
$maxlength The longest returned data length
$read_timeout The read operation timeout (requires PHP 4 Beta 4+)
Set to 0 for no timeout
$timed_out If a read operation times out, this attribute returns true (requires PHP 4 Beta 4+)
$maxframes The maximum number of frames allowed to be tracked
$status The status of the captured http
$temp_dir The web server can write Temporary file directory (/tmp)
$curl_path cURL binary directory, if there is no cURL binary, set it to false
Four. The following is the demo
include "Snoopy.class.php";
$snoopy = new Snoopy;
$snoopy- >proxy_host = "www.baidu.com";
$snoopy->proxy_port = "8080";
$snoopy->agent = "(compatible; MSIE 4.01; MSN 2.5; AOL 4.0; Windows 98)";
$snoopy->referer = "http://www.baidu.com/";
$snoopy->cookies["SessionID"] = 238472834723489l;
$snoopy->cookies["favoriteColor"] = "RED"; 
$snoopy->rawheaders["Pragma"] = "no-cache"; 
$snoopy->maxredirs = 2; 
$snoopy->offsiteok = false; 
$snoopy->expandlinks = false; 
$snoopy->user = "joe"; 
$snoopy->pass = "bloe"; 
if($snoopy->fetchtext("http://www.baidu.com")) 
{ 
echo "

".htmlspecialchars($snoopy->results)." 

n";
 
}
 
else
 
echo "error fetching document: ".$snoopy->error."n"; 
snoopy采集phpchina示例 
//采集phpchina 
set_time_limit(0); 
require_once("Snoopy.class.php"); 
$snoopy=new Snoopy(); 
//登陆论坛 
$submit_url = "http://www.phpchina.com/bbs/logging.php?action=login"; 
$submit_vars["loginmode"] = "normal"; 
$submit_vars["styleid"] = "1"; 
$submit_vars["cookietime"] = "315360000"; 
$submit_vars["loginfield"] = "username"; 
$submit_vars["username"] = "***"; //你的用户名 
$submit_vars["password"] = "*****"; //你的密码 
$submit_vars["questionid"] = "0"; 
$submit_vars["answer"] = ""; 
$submit_vars["loginsubmit"] = "提 交"; 
$snoopy->submit($submit_url,$submit_vars); 
if ($snoopy->results) 
{ 
//获取连接地址 
$snoopy->fetchlinks("http://www.phpchina.com/bbs"); 
$url=array(); 
$url=$snoopy->results; 
//print_r($url); 
foreach ($url as $key=>$value) 
{ 
//匹配http://www.phpchina.com/bbs/forumdisplay.php?fid=156&sid=VfcqTR地址即论坛板块地址 
if (!preg_match("/^(http://www.phpchina.com/bbs/forumdisplay.php?fid=)[0-9]*&sid=[a-zA-Z]{6}/i",$value)) 
{ 
unset($url[$key]); 
} 
} 
//print_r($url); 
//获取到板块数组$url，循环访问，此处获取第一个模块第一页的数据 
$i=0; 
foreach ($url as $key=>$value) 
{ 
if ($i>=1) 
{ 
//测试限制 
break; 
} 
else 
{ 
//访问该模块，提取帖子的连接地址，正式访问里需要提取帖子分页的数据，然后根据分页数据提取帖子数据 
$snoopy=new Snoopy(); 
$snoopy->fetchlinks($value); 
$tie=array(); 
$tie[$i]=$snoopy->results; 
//print_r($tie); 
//转换数组 
foreach ($tie[$i] as $key=>$value) 
{ 
//匹配http://www.phpchina.com/bbs/viewthread.php?tid=68127& extra=page%3D1&page=1&sid=iBLZfK 
if (!preg_match("/^(http://www.phpchina.com/bbs/viewthread.php?tid=)[0-9]*&extra=page%3D1&page=[0-9]*&sid=[a-zA-Z]{6}/i",$value)) 
{ 
unset($tie[$i][$key]); 
} 
} 
//print_r($tie[$i]); 
//归类数组，将同一个帖子不同页面的内容放一个数组里 
$left='';//连接左边公用地址 
$j=0; 
$page=array(); 
foreach ($tie[$i] as $key=>$value) 
{ 
$left=substr($value,0,52); 
$m=0; 
foreach ($tie[$i] as $pkey=>$pvalue) 
{ 
//重组数组 
if (substr($pvalue,0,52)==$left) 
{ 
$page[$j][$m]=$pvalue; 
$m++; 
} 
} 
$j++; 
} 
//去除重复项开始 
//$page=array_unique($page);只能用于一维数组 
$paget[0]=$page[0]; 
$nums=count($page); 
for ($n=1;$n <$nums;$n++) 
{ 
$paget[$n]=array_diff($page[$n],$page[$n-1]); 
} 
//去除多维数组重复值结束 
//去除数组空值 
unset($page); 
$page=array();//重新定义page数组 
$page=array_filter($paget); 
//print_r($page); 
$u=0; 
$title=array(); 
$content=array(); 
$temp=''; 
$tt=array(); 
foreach ($page as $key=>$value) 
{ 
//外围循环，针对一个帖子 
if (is_array($value)) 
{ 
foreach ($value as $k1=>$v1) 
{ 
//页内循环，针对一个帖子的N页 
$snoopy=new Snoopy();
$snoopy->fetch($v1);
$temp=$snoopy->results;
//Read title
if (!preg_match_all("/

( .*)

/i",$temp,$tt))
{
echo "no title";
exit;
}
else
{
$title[$u]=$tt[1] [1];
}
unset($tt);
//Read content
if (!preg_match_all("/ /i",$temp,$tt))
{
print_r($tt);
echo "no content1";
exit;
}
else
{
foreach ( $tt[1] as $c=>$c2)
{
$content[$u].=$c2;
}
}
}
}
else
{
//Get the page content directly
$snoopy =new Snoopy();
$snoopy->fetch($value);
$temp=$snoopy->results;
//Read title
if (!preg_match_all("/

(.* )

/i",$temp,$tt))
{
echo "no title";
exit;
}
else
{
$title[$u]=$tt[1][1 ];
}
unset($tt);
//Read content
if (!preg_match_all("/ /i",$temp,$tt))
{
echo "no content2";
exit;
}
else
{
foreach ($tt[1] as $c=>$c2 )
{
$content[$u].=$c2;
}
}
}
$u++;
}
print_r($content);
}
$i++;
}
}
else
{
echo "login failed";
exit;
}
?>