Selenium+PhantomJs parses and renders basic operations of Js

Some people say that the Selenium library and PhantomJ are a versatile tool when used together. So, are they really that powerful? Let’s take a look at the usage of the Selenium library. Through this article, let us take a look at the operation of the Selenium library combined with PhantomJs, Chrome and other browsers.

What is Selenium

Selenium is an automated testing tool that supports some browsers including Chrome, Firefox, Safari, PhantomJs, etc. If used in crawlers, we mainly use it to solve some JavaScript rendering problems.

When we use the Requests library to request some web pages, such as 163music, the response data we get is not all the information we see in the browser. He may be rendered through js. Then, if we use the Selenium library, we will no longer care about how to solve this problem.

Because our browser, such as PhantomJs, is an interfaceless browser. It is used to render and parse js, and the Selenium library is responsible for sending some commands to the browser and simulating some such as drop-down and dragging. , page turning, form input and other actions. In this way, the combination of the two of them can perfectly solve those JS rendering problems.

Note

Although the Selenium library plus PhantomJs is very useful, after all, it drives a browser and then obtains data. So when we use it, we will find that it is not as fast as some parsing libraries we use. This is actually its drawback, so I still suggest that you don’t use them until you really can’t find a solution.

Installation preparation

pip directly installs the Selenium library:

pip install selenium

Installation of browser driver:

Chrome browser driver

PhantomJs browser driver

We need to configure the installed browser driver to our environment variables. For Windows users, configuring environment variables is more troublesome. We need to find the location of the downloaded driver, then copy its file location and paste it into the environment variable.

Configuration is complete, enter the command line:

phantomjs -v

Check whether it is successful.

Usage Example

from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.common.keys import Keys from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.support.wait import WebDriverWait browser = webdriver.Chrome() try: browser.get('http://www.yukunweb.com') input = browser.find_element_by_id('s') input.send_keys('Python') input.send_keys(Keys.ENTER) wait = WebDriverWait(browser, 10) wait.until(EC.presence_of_element_located((By.ID, 'main'))) print(browser.current_url) print(browser.page_source) finally: browser.close()

If we run the above code, we will see that a Chrome browser is opened locally, and then I enter my Blog URL, then he will automatically enter 'Python' in the search bar and click Enter to search. And print out the URL and source code of the result page.

Our examples are all operated using the Chrome browser, because PhantomJs has no interface and it is inconvenient to see the effect. If you run it incorrectly, usually the browser is not open, then it should be that you have not installed the Chrome browser, or you have not configured the driver with environment variables.

So what do these lines of code mean, and what instructions do we give them?

Declare the browser object

from selenium import webdriver browser = webdriver.Chrome() # 声明其他浏览器 browser = webdriver.PhantomJs() browser = webdriver.Firefox()

This is equivalent to us calling the webdriver method of the Selenium library and instantiating a Chrome browser for us transfer.

Visit the page

from selenium import webdriver browser = webdriver.Chrome() browser.get('http://www.yukunweb.com')

We pass the url we want to visit to the get method. Call the browser to access the url.

Find elements

input = browser.find_element_by_id('s')

This code calls the find_element_by_id method. As the name suggests, it is to find the tag with the id of 's', then if it is an operation If class is 's', it is find_element_by_class('s').

Of course, we can also use CSS selectors and xpath selectors to find elements:

input = browser.find_element_by_css_selector("#s") print(input) input = browser.find_element_by_xpath('//*[@id="s"]') print(input)

By printing the results, you can see that no matter what selector is used, the search results are the same. The following are some search APIs:

find_element_by_namefind_element_by_xpathfind_element_by_link_textfind_element_by_partial_link_textfind_element_by_tag_namefind_element_by_class_namefind_element_by_css_selector

Find multiple elements

If the element we are looking for is the li tag in the web page, there are many elements. Then our search method is the same as for a single element, except that for the search API we need to add a plural form s after element. That is:

find_elements_by_namefind_elements_by_xpathfind_elements_by_link_textfind_elements_by_partial_link_textfind_elements_by_tag_namefind_elements_by_class_namefind_elements_by_css_selector

Element interactive operation

is to issue instructions and call interactive methods for the elements we obtain.

browser.get('http://www.yukunweb.com') input = browser.find_element_by_id('s') input.send_keys('Python') input.send_keys(Keys.ENTER)

In this code, we first find the element with id 's', then pass it the 'Python' value, then call the interactive method and type return car.

Of course, in most cases, we cannot directly use the method of hitting Enter, because we are not sure whether the form was submitted after hitting Enter. We need to use the finder to find the submit button element, and then simulate clicking:

button = browser.find_element_by_class_name('xxxx') button.click() # 清除表单信息 button.clear()

Then, we can see that when simulating login, we can directly enter the account number and password manually. If If there is a verification code, directly provide an input method. We manually enter the verification code and pass it to the form. Is it very simple to simulate login?

Interaction

元素交互动作与上面的操作是不同的。上面的操作需要获得一个特定的元素。然后对这个特定的元素调用一些指令，才可以完成交互。而这个交互是将这些动作附加到动作链中串行执行。

我们以拖拽元素为例(我们需要导入ACtionChains方法)：

from selenium import webdriver from selenium.webdriver import ActionChains browser = webdriver.Chrome() browser.get(url) source = browser.find_element_by_name("source") target = browser.find_element_by_name("target") actions = ActionChains(browser) actions.drag_and_drop(source, target).perform()

这里的sourcs是我们要拖拽的元素，我们使用查找器找到他，target就是我们要拖拽到的位置元素。然后调用ActionChains方法，实现拖拽操作。

执行JavaScript

有些动作呢，Selenium库并没有为我们提供特定的api，比如说将浏览器进度条下拉，这个实现起来是很难的。那么我们就可以通过让Selenium执行JS来实现进度条的下拉，这个得需要一些js的知识，不过还是很简单的。

from selenium import webdriver browser = webdriver.Chrome() browser.get('http://www.yukunweb.com') browser.execute_script('window.scrollTo(0, document.body.scrollHeight)') browser.execute_script('alert("到达底部")')

这就相当于我们将一些JS命令传给Selenium的execute_script这个api，我们运行就可以看到浏览器下拉到底部，然后弹出会话框。

获取元素文本值

如果我们查找得到一个元素，我们要怎样获得元素的一些属性和文本信息呢？

from selenium import webdriver browser = webdriver.Chrome() browser.get('http://www.yukunweb.com') name = browser.find_element_by_css_selector('#kratos-logo > a') print(name.text) print(name.get_attribute('href'))

运行结果可以看到，他打印出了‘意外’和他的url。

Frame框架

有些网页在我们直接使用Selenium驱动浏览器打印源码的时候，并没有如期获得想要的数据，那在我们查看网页源码的时候，可以看到网页的iframe标签包裹的一个一个的框架。那么这就需要我们请求对应框架，拿到源码了。

我们以网易云音乐的歌手栏为例。

from selenium import webdriver browser = webdriver.Chrome() browser.get('https://music.163.com/#/discover/artist/signed/') print(browser.page_source)

可以查看结果，并没有我们想要的信息。

from selenium import webdriver browser = webdriver.Chrome() browser.get('https://music.163.com/#/discover/artist/signed/') browser.switch_to.frame('contentFrame') print(browser.page_source)

这次打印，我们就可以看到我们需要的信息了，是不是很简单。

显示等待

在文章开始的时候，我们运行的那段代码中有一段代码是不是还没有说。那就是我们命令浏览器等待的操作。

等待有两种方式，一种是隐士等待，一种是显示等待。当使用了隐士等待执行时，如果浏览器没有找到指定元素，将继续等待，如果超出设定时间就会抛出找不到元素的异常。而大多数情况我们建议使用显示等待。

显示等待是你指定一个等待的条件，还指定一个最长等待时间。那么程序会在最长等待时间内，判断条件是否成立，如果成立，立即返回。如果不成立，他会一直等待，直到最长等待时间结束，如果条件仍然不满足，就返回异常。

wait = WebDriverWait(browser, 10) wait.until(EC.presence_of_element_located((By.ID, 'main')))

这里的By.ID方法实际上就是一个查找的万能方法，而我们直接查找或者使用CSS、xpath查找足够满足，我也不过多介绍，想要了解可以查看官方文档。

这里是知道查找到id为‘main’就返回。

显示等待的一些条件还有：

title_is 标题是某内容

title_contains 标题包含某内容

presence_of_element_located 元素加载出，传入定位元组，如(By.ID, ‘p’)

visibility_of_element_located 元素可见，传入定位元组

visibility_of_element_located 元素可见，传入定位元组

visibility_of_element_located 元素可见，传入定位元组

visibility_of 可见，传入元素对象

presence_of_all_elements_located 所有元素加载出

text_to_be_present_in_element 某个元素文本包含某文字

text_to_be_present_in_element_value 某个元素值包含某文字

frame_to_be_available_and_switch_to_it frame加载并切换

invisibility_of_element_located 元素不可见

element_to_be_clickable 元素可点击

staleness_of 判断一个元素是否仍在DOM，可判断页面是否已经刷新

element_to_be_selected 元素可选择，传元素对象

element_located_to_be_selected 元素可选择，传入定位元组

element_selection_state_to_be 传入元素对象以及状态，相等返回True，否则返回False

element_located_selection_state_to_be 传入定位元组以及状态，相等返回True，否则返回False

alert_is_present 是否出现Alert

窗口选择

如果我们在表单输入关键词，提交表单后浏览器新打开了一个窗口，那么我们要怎么去操作新的窗口呢？索性Selenium为我们提供了对应的api.

import time from selenium import webdriver from selenium.webdriver.common.keys import Keys browser = webdriver.Chrome() browser.get('http://www.23us.cc/') input = browser.find_element_by_id('bdcs-search-form-input') input.send_keys('斗破苍穹') input.send_keys(Keys.ENTER) browser.switch_to_window(browser.window_handles[1]) print(browser.current_url) time.sleep(1) browser.switch_to_window(browser.window_handles[0]) print(browser.current_url)

通过打印结果，不难看出先打印了搜索结果窗口url，然后打印了索引页url。要注意窗口的索引是从 0 开始的哦，这个大家都明白。

异常处理

异常处理和普通的异常处理一样，没有什么要说的，大家自己查看官方异常 api.地址

最后

Okay, through this article, I hope you can basically understand some of the ways to use the Selenium library combined with browser drivers. We use Chrome in our example, but in actual code it is best to use PhantomJs, because it has no interface and runs relatively better.

I said at the beginning of the article that it is generally not recommended that you use Selenium because it is very slow. But even if it’s slow, it’s still very cool, isn’t it?

The above is the detailed content of Selenium+PhantomJs parses and renders basic operations of Js. For more information, please follow other related articles on the PHP Chinese website!