Teach you step-by-step how to use Python web crawler to obtain the video selection content of Bilibili (source code attached)

1. Background introduction

When it comes to Bilibili, the first impression is the video. I believe that many friends, like me, want to use web crawler technology Get the videos from Station B, but the videos from Station B are actually not that easy to get.Guanis about getting the videos from Station B, which was previously available atThe introduction is implemented through theyou-get library.Interested friends can read this article:You-Get is so powerful!.

## Closer to home, friends who often study on Bilibili may often encounter bloggers who serialize dozens or even hundreds of videos, especially like this one For continuous tutorials on programming languages, courses, tool usage, etc., a selection series will appear, as shown in the figure below.

Of coursethe fields of theseselectionscan also be seen by our naked eyes.JustIf it is implemented through a program, it may not be as simple as imagined. So the goal of this article is to obtain video selections through Python web crawler technology and based on the selenium library.

2. Specific implementation

The library we use in this article is selenium. This is a Although the library used to simulate user login feels slow, it is still used quite a lot in the field of web crawlers. It has been used repeatedly to simulate login and obtain data. Below is all the code to implement video selection collection. You are welcome to practice it yourself.

# coding: utf-8 from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.support.wait import WebDriverWait class Item: page_num = "" part = "" duration = "" def __init__(self, page_num, part, duration): self.page_num = page_num self.part = part self.duration = duration def get_second(self): str_list = self.duration.split(":") sum = 0 for i, item in enumerate(str_list): sum += pow(60, len(str_list) - i - 1) * int(item) return sum def get_bilili_page_items(url): options = webdriver.ChromeOptions() options.add_argument('--headless') # 设置无界面 options.add_experimental_option('excludeSwitches', ['enable-automation']) # options.add_experimental_option("prefs", {"profile.managed_default_content_settings.images": 2, # "profile.managed_default_content_settings.flash": 0}) browser = webdriver.Chrome(options=options) # browser = webdriver.PhantomJS() print("正在打开网页...") browser.get(url) print("等待网页响应...") # 需要等一下，直到页面加载完成 wait = WebDriverWait(browser, 10) wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@class="list-box"]/li/a'))) print("正在获取网页数据...") list = browser.find_elements_by_xpath('//*[@class="list-box"]/li') # print(list) itemList = [] second_sum = 0 # 2.循环遍历出每一条搜索结果的标题 for t in list: # print("t text:",t.text) element = t.find_element_by_tag_name('a') # print("a text:",element.text) arr = element.text.split('\n') print(" ".join(arr)) item = Item(arr[0], arr[1], arr[2]) second_sum += item.get_second() itemList.append(item) print("总数量:", len(itemList)) # browser.page_source print("总时长/分钟:", round(second_sum / 60, 2)) print("总时长/小时:", round(second_sum / 3600.0, 2)) browser.close() return itemList get_bilili_page_items("https://www.bilibili.com/video/BV1Eb411u7Fw")

The selector used here is xpath, and the video example is the Tongji Edition of "Advanced Mathematics" at Station B Full teaching video (Teacher Song Hao) video selection. If you want to grab other video selections, you only need to change the URL link in the last line of the above code.

三、常见问题

在运行过程中小伙伴们应该会经常遇到这个问题，如下图所示。

这个是因为谷歌驱动版本问题导致的，只需要根据提示，去下载对应的驱动版本即可，驱动下载链接：

https://chromedriver.storage.googleapis.com/index.html

The above is the detailed content of Teach you step-by-step how to use Python web crawler to obtain the video selection content of Bilibili (source code attached). For more information, please follow other related articles on the PHP Chinese website!