Solution to the problem of crawling web page data with Puppeteer to return empty arrays

This article aims to solve the problem of returning an empty array when crawling web page data using Puppeteer. By analyzing common reasons and providing optimized code examples, we can help developers crawl the data of the target website more effectively and avoid the crawling results when the crawling result is empty. This article will focus on key links such as selector optimization, page element loading, and data extraction.

Problem analysis

When using Puppeteer for web page data crawling, there are usually several reasons to return an empty array:

1. Selector Error: The CSS selector or XPath expression is incorrect, causing the target element to be unable to be found.
2. Page not fully loaded: When performing a crawl operation, the page may not have been fully loaded, resulting in the element not present.
3. Dynamic content: The target data is loaded dynamically through JavaScript, and Puppeteer needs to wait for the data to be loaded.
4. Elements are removed or hidden: The target element is removed or hidden before being crawled, resulting in data not being retrieved.
5. Loop logic error: When looping through elements, index or condition judgment is wrong, resulting in not correctly extracting all data.

Solution

In response to the above issues, the following measures can be taken:

1. Optimized selector: Use more precise selectors to ensure uniquely positioning of the target element. You can use the browser's developer tools to assist in writing selectors.
2. Wait for the page to load: Use page.waitForSelector() or page.waitForTimeout() and other methods to ensure that the page element is loaded before performing the crawling operation.
3. Process dynamic content: Use page.waitForFunction() and other methods to wait for the dynamic data to load.
4. Check whether the element exists: Before grabbing the element, use the page.$() method to check whether the element exists to avoid errors caused by the element's non-existence.
5. Optimize loop logic: carefully check the loop's index and conditional judgment to ensure that all target elements can be correctly traversed.

Code Example

Here is an optimized Puppeteer code example to crawl baby names and meanings on web pages.

 const puppeteer = require("puppeteer");
const express = require("express");
const cors = require("cors");
const app = express();
app.use(cors());
let data = [];

(async () => {
  const browser = await puppeteer.launch({
    headless: true,
    defaultViewport: null,
  });
  const page = await browser.newPage();

  for (let pageNumber = 1; pageNumber  i`);

    // Loop through the element for (let i = 0; i  el.textContent, nameElements[i]);
      let meaning = await page.evaluate(el => el.textContent, meaningElements[i]);
      fullName = `${name.split(/[\n\t]/).join('').trim()}, ${meaning}`;
      data.push({ fullName });
    }
  }

  console.log(data);
  await browser.close();
})();

app.get("/", (req, res) => {
  res.status(200).json(data);
});

app.listen(3000, () => {
  console.log("App is running...");
});

Code explanation:

• Selector optimization: Use a.nsg__name and div.nsg__meaning > i to locate the name and meaning elements more accurately.
• Remove unnecessary click action: Remove click popup action because it has nothing to do with data crawling.
• Loop traversal: Use for loop to loop through all the name and meaning elements and combine them into complete data.
• Text processing: Use split(/[\n\t]/).join('').trim() to clean text data and remove line breaks, tabs and spaces.

Things to note

• Website anti-climbing mechanism: Some websites may adopt anti-climbing mechanisms, such as verification codes, IP restrictions, etc. It is necessary to take corresponding countermeasures based on actual conditions, such as using proxy IP, setting up User-Agent, etc.
• Comply with website rules: When crawling web page data, you should abide by the website's Robots.txt protocol to avoid excessive crawling to avoid burdening the website.
• Data cleaning: The captured data may contain noise and need to be cleaned and processed to obtain effective information.

Summarize

By optimizing selectors, waiting for page loading, processing dynamic content, checking whether elements exist, and optimizing loop logic, the problem of Puppeteer crawling web page data and returning empty arrays can be effectively solved. In actual applications, adjustments and optimizations need to be made according to specific circumstances in order to achieve ideal crawling results.

The above is the detailed content of Solution to the problem of crawling web page data with Puppeteer to return empty arrays. For more information, please follow other related articles on the PHP Chinese website!