Home  >  Article  >  Backend Development  >  Python crawler---Autohome font anti-crawling

Python crawler---Autohome font anti-crawling

云罗郡主
云罗郡主Original
2019-01-21 10:58:064707browse





##This article brings you the content about Python crawler---Autohome font reverse crawling has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

Target website: Autohome

Target website: https://club.autohome.com.cn/bbs/thread/2d8a42404ba24266/77486027-1.html#pvareaid=2199101

Reason:

Today my old colleague asked me to look at a website: "Auto Home". At first I looked at it and felt that there should be no anti-crawling measures, but during the actual operation, I found that it turned out to be a font Anti-crawling encryption is performed.

View source code:

Python crawler---Autohome font anti-crawling

Can’t help but be surprised, is it UTF-8 encoded text? So I converted online,

Python crawler---Autohome font anti-crawling

#After converting, I still found that the font I got was still a fake font. Not believing in evil, I pasted the article into NotePad again and found:

Python crawler---Autohome font anti-crawling

It turns out that these fonts cannot be pasted. Well, we can only consider that the font settings are anti-crawling. I saw

in the source code, so I looked through myfont to see if I could find anything? As a result, I really found something

Isn’t this a ..ttf font? Is it encrypted using this font? Then download this font first to see if the guess is correct. After downloading the font, here is a tool for viewing ttf fonts: Font Creator (download by yourself...) Download and open to view

Python crawler---Autohome font anti-crawling

Each character above shows its glyph and its glyph code, so I can’t help but think about it, how to convert these two fonts? Looking through many big blogs, I found a python package that specializes in analyzing fonts, fonttools. Install pip install fonttools by yourself. Reference link: https://darknode.in/font/font-tools-guide/

# 那么便开始通过字体库进行解析
world = TTFont('./world.ttf')
# 读取响应的映射关系
uni_list = world['cmap'].tables[0].ttFont.getGlyphOrder()  # 'cmap' 表示汉字对应的映射 为unicode编码
print(uni_list) # 按顺序拿到各个字符的unicode编码
# 打印结果:
['.notdef', 'uniEDE8', 'uniED35', 'uniED87', 'uniECD3', 'uniED25', 'uniEC72', 'uniEDB2', 'uniEE04', 'uniED51', 'uniEC9D', 'uniECEF', 'uniEC3C', 'uniEC8D', 'uniEDCE', 'uniED1B', 'uniED6C', 'uniECB9', 'uniEDFA', 'uniEC57', 'uniED98', 'uniEDEA', 'uniED36', 'uniEC83', 'uniECD5', 'uniEC21', 'uniED62', 'uniEDB4', 'uniED00', 'uniED52', 'uniEC9F', 'uniEDDF', 'uniEC3D', 'uniED7E', 'uniECCA', 'uniED1C', 'uniEC69', 'uniECBB', 'uniEDFB']
# .notdef 并不是汉字的映射, 而是表示字体家族名称。
将映射列表转换成unicode的类型,因为自己文中获取的是字符串unicode类型的,当然你也可以转化为utf-8,不过你获取的文章内容也要转化为utf-8
unicode_list= [eval(r"u'\u" + uni[3:] + "'") for uni in uni_list[1:]]

Regarding mapping, I actually have some doubts about what is between utf-8 and unicode. relation? It is recommended to refer to Zhihu for detailed explanation:

https://www.zhihu.com/question/23374078

Okay, everything has been prepared above. Let’s write some code.

# coding:utf-8
import re
import requests
from scrapy import Selector
from fontTools.ttLib import TTFont
class QiCheZhiJiaSpider:
    def article_content(self):
        url = 'https://club.autohome.com.cn/bbs/thread/2d8a42404ba24266/77486027-1.html#pvareaid=2199101'
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36'
        }
        try:
            response = requests.get(url=url, headers=headers).text
            response_info = Selector(text=response)
        except BaseException as e:
            print(e)
        else:
            content = response_info.xpath('//div[@class="tz-paragraph"]//text()').extract()  # 获取列表的形式内容。
            # print(content)
            content_str = ''.join(content)
            # 紧接着获取字体的链接
            world_href = re.findall(r",url\('(//.*\.ttf)'\).*", response, re.M or re.S)[0]
            world_href = 'https:' + world_href
            world_content = requests.get(url=world_href, headers=headers).content
            # 对获取到的字体进行下载..........
            with open('./world.ttf', 'wb') as f:
                f.write(world_content)
            # 那么便开始通过字体库进行解析
            world = TTFont('./world.ttf')
            # 读取响应的映射关系
            uni_list = world['cmap'].tables[0].ttFont.getGlyphOrder()
            unicode_list = [eval(r"u'\u" + uni[3:] + "'") for uni in uni_list[1:]]
            world_list = ["右", "远", "高", "呢", "了", "短", "得", "矮", "多", "二", "大", "一", "不", "近",
                          "是", "着", "五", "三", "九", "六", "少", "好", "上", "七", "和", "很", "十",
                          "四", "左", "下", "八", "小", "坏", "低", "长", "更", "的", "地"]  # # 录入字体文件中的字符。必须要以国际标准的unicode编码
            for i in range(len(unicode_list )):
                content_str = content_str.replace(unicode_list [i], world_list[i])
            print(content_str)
if __name__ == '__main__':
    qi_che_zhi_jia = QiCheZhiJiaSpider()
    qi_che_zhi_jia.article_content()

Result display:

Python crawler---Autohome font anti-crawling

The above is all the introduction, I hope everyone can gain something, more

Python video tutorial Please pay attention to the PHP Chinese website.




##

The above is the detailed content of Python crawler---Autohome font anti-crawling. For more information, please follow other related articles on the PHP Chinese website!

Statement:
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn