Process dynamic web content in R language: identify and use API to obtain data

This tutorial aims to solve the challenges encountered when crawling dynamically loading web content using the R language rvest package. When traditional HTML parsing methods cannot obtain the data rendered by JavaScript, the core strategy is to identify the API interface called behind the web page. We will demonstrate how to efficiently and accurately extract the required information by directly requesting these APIs and parsing the JSON data they return, thereby overcoming the limitations of front-end dynamic rendering.

In modern web design, JavaScript is widely used to load and render content dynamically, which means that most of the data users see in the browser is not directly embedded in the initially loaded HTML document. When trying to crawl such websites using traditional HTML parsing tools such as rvest, you often encounter problems that you cannot get the expected data, such as returning empty character vectors. This is usually due to the fact that the target data is fetched from the backend API via an asynchronous JavaScript call (AJAX or Fetch API) and then dynamically inserted into the page DOM on the browser side.

Limitations of traditional HTML parsing

When the HTML document loaded by rvest::read_html does not contain the specific data you are looking for (such as nested links), it indicates that the content is likely to be dynamically generated through client JavaScript after the page is loaded. In this case, it is invalid to just parse the static HTML. For example, the following code tries to extract the link from the Thrive Market page, but gets an empty result:

 library(rvest)
library(xml2)

page1_url %
# rvest::html_nodes('body') %>%
# xml2::xml_find_all("//div[contains(@class, 'd85qmy-0 kRbsKs')]") %>%
# rvest::html_attr('href')
# print(page1_urls) # The result is usually empty

This indicates that the link we are looking for does not exist in the HTML source code originally obtained by read_html.

Identify API sources for dynamic content

The key to solving this problem is to identify the actual source of the data—usually the backend API of the website. We can observe network requests through the browser's developer tools (usually pressing F12 to open).

1. Open Developer Tools: On the landing page, press F12 to open Developer Tools.
2. Switch to the Network tab: This displays all HTTP requests made when the browser loads the page.
3. Reload the page: After clearing the network request list, refresh the page to capture all requests.
4. Filter and check requests: Focus on requests of XHR (XMLHttpRequest) or Fetch type. These requests are usually used to load data asynchronously. Check their URL, request method (GET/POST), and response content. The response content is usually in JSON or XML format, containing data displayed dynamically on the page.

In this way, we can find that many dynamically loaded product lists or details are actually obtained through a specific API endpoint, such as https://thrivemarket.com/api/v1/products.

Use API to directly obtain data

Once the API endpoint is identified, we can bypass the front-end JavaScript rendering, directly send an HTTP request to the API and parse the data it returns. This approach is usually more efficient and stable than emulating browser behavior.

Step 1: Build the API request URL

Based on the API request information captured in the developer tool, we can build the corresponding API URL. These URLs usually contain query parameters, which are used to control paging, filtering conditions, etc. For example, the product data API of Thrive Market may contain parameters such as page_size and cur_page:

 # Build API request URL
api_url <h4> Step 2: Send HTTP request and parse JSON response</h4><p> The http package is a powerful tool in R language for sending HTTP requests. httr::GET() is used to send GET requests, while the httr::content() function can intelligently parse response content, especially JSON data.</p><pre class="brush:php;toolbar:false"> library(httr)
library(dplyr)
library(jsonlite) # explicitly load jsonlite to process JSON data# Build API request URL
api_url  0) {
    product_data %
      bind_rows() %>%
      as_tibble()

    print(product_data)
  } else {
    message("Product data not found or JSON structure does not meet expectations.")
  }
} else {
  message(paste("API request failed or returns non-JSON data. Status code:", status_code(response)))
}

Sample output (part):

 # A tibble: 60 x 2
   product url                         
   <chr> <chr>                       
 1 Organic Extra Virgin Olive Oil https://thrivemarket.com/p/~
 2 Grass-Fed Collagen Peptides https://thrivemarket.com/p/~
 3 Grass-Fed Beef Sticks, Original https://thrivemarket.com/p/~
 4 Organic Dry Roasted & Salted Cashews https://thrivemarket.com/p/~
 5 Organic Vanilla Extract https://thrivemarket.com/p/~
 6 Organic Raw Cashews https://thrivemarket.com/p/~
 7 Organic Coconut Milk, Regular https://thrivemarket.com/p/~
 8 Organic Robust Maple Syrup, Grade A, Value Size https://thrivemarket.com/p/~
 9 Organic Coconut Water https://thrivemarket.com/p/~
10 Non-GMO Avocado Oil Potato Chips, Himalayan Salt https://thrivemarket.com/p/~
# ... with 50 more rows</chr></chr>

With this approach, we successfully extract the product name and corresponding URL from the dynamically loaded web page without having to deal with complex JavaScript renderings.

Things to note

• API Structure and Stability: The API structure of a website may change over time. If your data crawling script suddenly fails, first check if the API endpoint URL or JSON response structure has been updated.
• Request frequency and limitations: Most APIs have request frequency limits (Rate Limit). Frequent or excessive requests may result in your IP being temporarily or permanently banned, or API access is restricted. Be sure to comply with the robots.txt file and terms of use of the website.
• Authentication and authorization: Some APIs require authentication (such as API Key, OAuth token). In this case, you need to include the corresponding authentication information in the HTTP request header. httpr::add_headers() can help you add this information.
• Error handling: In actual applications, robust error handling mechanisms should be added, such as checking HTTP response status code (response$status_code), handling network connection errors or parsing JSON failures.
• Data analysis complexity: The structure of JSON data may be very complex, and it is necessary to carefully analyze its nesting hierarchy and flexibly use tools such as lapply, purrr::map series functions or jsonlite::flatten() for recursive parsing.
• Law and Ethics: When conducting any data crawling activities, be sure to comply with the target website's terms of service, privacy policies, and local laws and regulations.

Summarize

When traditional HTML parsing-based web crawling methods (such as rvest) encounter bottlenecks when dealing with dynamic loading of content, identifying and directly interacting with the website's backend API is a powerful and efficient alternative. By analyzing network requests with the browser's developer tools, we can find the API endpoint where the data comes from, and then use http to send the HTTP request and parse the returned JSON data. This method not only allows you to obtain data more accurately, but also improves the efficiency and stability of the crawling. It is an important skill in R language for advanced web data acquisition. However, always remember to abide by the website’s usage regulations and ethics.

The above is the detailed content of Process dynamic web content in R language: identify and use API to obtain data. For more information, please follow other related articles on the PHP Chinese website!