Problems with python lxml parsing Chinese
高洛峰
高洛峰 2017-05-18 10:44:31
0
2
600

Using lxml to capture Chinese characters, the result is very painful, I don’t know how to deal with it...

comUrl="http://m.51job.com/search/codetail.php?coid=4108723"
res=requests.get(comUrl)
html=etree.HTML(res.text)
p=html.xpath("//aside")[1].xpath("./p")   #结果为[<Element p at 0x7bf01c8>, <Element p at 0x78f4408>, <Element p at 0x69db388>]
p[0].xpath("./span/text()")  #这个是想要抓取的字符

The result is captured like this [u'\xe6\x80\xa7\xe8\xb4\xa8']
unicode but the content is str encoding, how to convert this thing into Chinese?
Normally it should be '\xe6\x80\xa7\xe8\xb4\xa8' or u'\u6027\u8d28'

高洛峰
高洛峰

拥有18年软件开发和IT教学经验。曾任多家上市公司技术总监、架构师、项目经理、高级软件工程师等职务。 网络人气名人讲师,...

reply all(2)
淡淡烟草味
''.join(map(lambda x:chr(x), map(lambda x:ord(x), u'\xe6\x80\xa7\xe8\xb4\xa8'))).decode('utf-8')
滿天的星座

When this happens, it’s usually because requests have guessed the wrong encoding of the web page
So just specify the encoding of requests.
res.encoding ='utf-8'

In [33]: comUrl="http://m.51job.com/search/codetail.php?coid=4108723"
    ...: res=requests.get(comUrl)
    ...: res.encoding ='utf-8'
    ...: html=etree.HTML(res.text)
    ...: p=html.xpath("//aside")[1].xpath("./p")   #结果为[<Element p at 0x7b
    ...: f01c8>, <Element p at 0x78f4408>, <Element p at 0x69db388>]
    ...: p[0].xpath("./span/text()")  #这个是想要抓取的字符
    ...: 
Out[33]: [u'\u6027\u8d28']

In [34]: print _[0]
性质
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template