How to make a simple search engine

Sometimes because of work and our own needs, we will browse different websites to obtain the data we need, so crawlers come into being. The following is my process of developing a simple crawler and the problems I encountered.

Last time, Xiaobai had got a hard-working web crawler. It seemed that he would be sorry if he didn’t cause a series of small things, so Xiaobai started tinkering with information from various experts. Inverted index is a simple search engine designed based on basic principles.

The previous crawler only got the source code of the web page without doing any processing. It was a one-time small crawler, so Xiaobai used regular expressions to match the content of the web page to get the URL, and then the small crawler We can use this to help us crawl web pages until death. I have to mention beautifulsoup and regular expressions here. It is said that the beautifulsoup module is a powerful tool for web crawling and extraction. It is a pity that Xiaobao is not finished. I heard the name later and regretted not being able to try it out. However, Xiaobai has personally studied the regular expressions. Once he is proficient (forced proficiency), it is also very easy to use. For example, the URL for extracting the source code of the web page:

    link_list = re.findall(r"(?<=href=\").+?(?=\")|(?<=href=\&#39;).+?(?=\&#39;)", html)

    这一句就提出个七七八八来，当然这么粗糙鱼目混珠的情况也是少不了的，但是还是非常好用的，
    虽然看起来很复杂但是只要掌握了(?<)、(?=)、.、+、?这几种符号的用法小白觉得就可以解决大部分问题了，
    哦这里还有提一句，正则表达式似乎不支持嵌套的情形，
    大概形如“找到所有前面三个字符满足条件A的字符a,条件A是这三个字符前面的内容满足条件B”巴拉巴拉，
    好吧感觉说的好乱让我们暂且跳过这个话题。提取处理的匹配如果用findall是存储在列表中的，
    这样我们就可以在一个网页列表中一直加入新找到的链接一直重复爬取，小白这里自认为能力有限，
    所以就从提取中的网页链接爬取了100个网页做一个轻装版。这里提个醒，由于之前html链接提取的很粗糙，
    所以可能把各种形如网页链接的css文件路径、图片路径什么乱七八糟的给爬下来，
    不过我们这里就统一当作网页链接在一个try·except模块中进行connection，
    不是链接的就会出现异常我们只要捕获跳过就又可以愉快的继续了~~。

    有了爬取下来的网页内容下面就应该是得到其中真正呈现在网页中的东西了。
    写过网页的同学们都应该知道网页内容一般都在<p><\p>之中，title和链接什么的也有对应的标签，
    运用正则表单式理论上可以分离出来，不过小白亲身时间发现只匹配一次效果非常不好，
    匹配的内容的确包括想要的内容，但是因为标签一般都是嵌套的嘛而且小白技术毕竟也不好正则表达式可能表述的也有问题，
    所以总是会将内容嵌套在标签中返回，这里就有一个比较笨的方法供大家参考，咳咳，
    既然一次不能得到，那么就对内容进行再匹配，咳咳，经过了三层匹配外加一些小技巧终于是勉强匹配出来了，
    这里代码过于丑陋就不再贴出来了咳咳。

Related recommendations:

Scrapy crawler introductory tutorial four Spider (crawler)

php realizes the development of simple crawlers, PHP implements crawler

The above is the detailed content of How to make a simple search engine. For more information, please follow other related articles on the PHP Chinese website!