Python crawling method of Anjuke second-hand housing website data sharing

This article mainly brings you a python crawling of Anjuke second-hand housing website data (explanation with examples). The editor thinks it’s pretty good, so I’ll share it with you now and give it as a reference. Let’s follow the editor to take a look, I hope it can help everyone.

Now we will start to officially write the crawler. First, we need to analyze the structure of the website to be crawled: As a student in Henan, let’s take a look at the second-hand housing information in Zhengzhou!

In the above page, we can see the property information one by one. From the above, we can see the property information one by one on the web page. After clicking in, you will find:

Details of the property. OK! So what are we going to do? That is to get all the second-hand housing information in Zhengzhou and save it in the database. What is it used for? As a geographer, it is still somewhat useful. I won’t go into it this time. Okay, let’s officially start. First, I use the requests and BeautifulSoup modules in python3.6 to crawl the page. First, the requests module makes the request:

# 网页的请求头
header = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
}
# url链接
url = 'https://zhengzhou.anjuke.com/sale/'
response = requests.get(url, headers=header)
print(response.text)

After execution, You will get the html code of this website

Through analysis, you can get that each house is in the li tag of class="list-item", then we can proceed based on the BeautifulSoup package Extracting

# 通过BeautifulSoup进行解析出每个房源详细列表并进行打印
soup = BeautifulSoup(response.text, 'html.parser')
result_li = soup.find_all('li', {'class': 'list-item'})
for i in result_li:
  print(i)

can further reduce the amount of code by printing. OK, continue to extract

# 通过BeautifulSoup进行解析出每个房源详细列表并进行打印
soup = BeautifulSoup(response.text, 'html.parser')
result_li = soup.find_all('li', {'class': 'list-item'})
# 进行循环遍历其中的房源详细列表
for i in result_li:
  # 由于BeautifulSoup传入的必须为字符串，所以进行转换
  page_url = str(i)
  soup = BeautifulSoup(page_url, 'html.parser')
  # 由于通过class解析的为一个列表，所以只需要第一个参数
  result_href = soup.find_all('a', {'class': 'houseListTitle'})[0]
  print(result_href.attrs['href'])

. In this way, we You can see the URLs one by one. Do you like it?

Okay, according to normal logic, you have to enter the page and start analyzing the detailed page, but how to proceed to the next page after crawling? So, we need to first analyze whether the page has a next page

The same method can be used to find that the next page is also so simple, then we You can continue with the original recipe and original taste

# 进行下一页的爬取
result_next_page = soup.find_all('a', {'class': 'aNxt'})
if len(result_next_page) != 0:
  print(result_next_page[0].attrs['href'])
else:
  print('没有下一页了')

Because when the next page exists, there is an a tag in the web page. If not, it will become i tag, so this will do. Therefore, we can improve it and encapsulate the above into a function

import requests
from bs4 import BeautifulSoup

# 网页的请求头
header = {
  'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
}

def get_page(url):
  response = requests.get(url, headers=header)

  # 通过BeautifulSoup进行解析出每个房源详细列表并进行打印
  soup = BeautifulSoup(response.text, 'html.parser')
  result_li = soup.find_all('li', {'class': 'list-item'})

  # 进行下一页的爬取
  result_next_page = soup.find_all('a', {'class': 'aNxt'})
  if len(result_next_page) != 0:
    # 函数进行递归
    get_page(result_next_page[0].attrs['href'])
  else:
    print('没有下一页了')

  # 进行循环遍历其中的房源详细列表
  for i in result_li:
    # 由于BeautifulSoup传入的必须为字符串，所以进行转换
    page_url = str(i)
    soup = BeautifulSoup(page_url, 'html.parser')
    # 由于通过class解析的为一个列表，所以只需要第一个参数
    result_href = soup.find_all('a', {'class': 'houseListTitle'})[0]
    # 先不做分析，等一会进行详细页面函数完成后进行调用
    print(result_href.attrs['href'])


if __name__ == '__main__':
  # url链接
  url = 'https://zhengzhou.anjuke.com/sale/'
  # 页面爬取函数调用
  get_page(url)

Okay, then let’s start the detailed page Crawled

Hey, the power is always cut off, what a trap in the university, I will attach the results first, I will add more when I have free time,

import requests
from bs4 import BeautifulSoup

# 网页的请求头
header = {
  'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
}

def get_page(url):
  response = requests.get(url, headers=header)

  # 通过BeautifulSoup进行解析出每个房源详细列表并进行打印
  soup_idex = BeautifulSoup(response.text, 'html.parser')
  result_li = soup_idex.find_all('li', {'class': 'list-item'})

  # 进行循环遍历其中的房源详细列表
  for i in result_li:
    # 由于BeautifulSoup传入的必须为字符串，所以进行转换
    page_url = str(i)
    soup = BeautifulSoup(page_url, 'html.parser')
    # 由于通过class解析的为一个列表，所以只需要第一个参数
    result_href = soup.find_all('a', {'class': 'houseListTitle'})[0]
    # 详细页面的函数调用
    get_page_detail(result_href.attrs['href'])


  # 进行下一页的爬取
  result_next_page = soup_idex.find_all('a', {'class': 'aNxt'})
  if len(result_next_page) != 0:
    # 函数进行递归
    get_page(result_next_page[0].attrs['href'])
  else:
    print('没有下一页了')

# 进行字符串中空格，换行，tab键的替换及删除字符串两边的空格删除
def my_strip(s):
  return str(s).replace(" ", "").replace("\n", "").replace("\t", "").strip()
# 由于频繁进行BeautifulSoup的使用，封装一下，很鸡肋
def my_Beautifulsoup(response):
  return BeautifulSoup(str(response), 'html.parser')



# 详细页面的爬取
def get_page_detail(url):
  response = requests.get(url, headers=header)
  if response.status_code == 200:
    soup = BeautifulSoup(response.text, 'html.parser')
    # 标题什么的一大堆，哈哈
    result_title = soup.find_all('h3', {'class': 'long-title'})[0]
    result_price = soup.find_all('span', {'class': 'light info-tag'})[0]
    result_house_1 = soup.find_all('p', {'class': 'first-col detail-col'})
    result_house_2 = soup.find_all('p', {'class': 'second-col detail-col'})
    result_house_3 = soup.find_all('p', {'class': 'third-col detail-col'})
    soup_1 = my_Beautifulsoup(result_house_1)
    soup_2 = my_Beautifulsoup(result_house_2)
    soup_3 = my_Beautifulsoup(result_house_3)
    result_house_tar_1 = soup_1.find_all('dd')
    result_house_tar_2 = soup_2.find_all('dd')
    result_house_tar_3 = soup_3.find_all('dd')
    '''
    文博公寓，省实验中学，首付只需70万，大三房，诚心卖，价可谈 270万
    宇泰文博公寓 金水－花园路－文博东路4号 2010年 普通住宅
    3室2厅2卫 140平方米 南北 中层(共32层)
    精装修 19285元/m² 81.00万
    '''
    print(my_strip(result_title.text), my_strip(result_price.text))
    print(my_strip(result_house_tar_1[0].text),
       my_strip(my_Beautifulsoup(result_house_tar_1[1]).find_all('p')[0].text),
       my_strip(result_house_tar_1[2].text), my_strip(result_house_tar_1[3].text))
    print(my_strip(result_house_tar_2[0].text), my_strip(result_house_tar_2[1].text),
       my_strip(result_house_tar_2[2].text), my_strip(result_house_tar_2[3].text))
    print(my_strip(result_house_tar_3[0].text), my_strip(result_house_tar_3[1].text),
       my_strip(result_house_tar_3[2].text))

if __name__ == '__main__':
  # url链接
  url = 'https://zhengzhou.anjuke.com/sale/'
  # 页面爬取函数调用
  get_page(url)

Since I wrote the code while blogging, I made some changes in the get_page function, that is, the recursive call for the next page needs to be placed after the function, and the two functions are encapsulated without introduction,

And the data is not written to mysql, so I will continue to follow up later, thank you!!!

Related recommendations:

python crawling article example tutorial

10 recommended articles about python crawling

Share a Python method to crawl popular comments on NetEase Cloud Music

The above is the detailed content of Python crawling method of Anjuke second-hand housing website data sharing. For more information, please follow other related articles on the PHP Chinese website!