Asksearch_php.cn

Community

Learn

Tools Library

AI Tools

Leisure

English

Popular searches：
PHP
MySQL
jquery
HTML
CSS

How to prevent malicious ddos crawling on nginx

First of all, I have no objection to others crawling the content of my website. I don’t necessarily strictly limit other people’s crawling. However, some people’s crawling has no bottom line at all. They use one script or even multiple scripts to crawl a certain website concurrently. The content of ...

黄舟 2017-05-16 17:30:17

python - pyspider scheduled crawling problem

When writing the crawler, I found that after setting every in the code, after crawling once on the 21st, I saw that the result was not updated today, and the lastcrawltime was still on the 21st. Is my parameter setting incorrect?

我想大声告诉你 2017-05-18 10:53:29

javascript - Problem with crawling web page Jquery selector first-child

When crawling a website, I feel that h2 and h3 have the same structure. Why can h2:first-child get data, but h3 cannot? The final results h2_1 and h2_2 are the same, no problem. h3_1 is OK, but h3_2 is empty. Why is this? The code is as follows, {code...}

巴扎黑 2017-05-16 13:28:41

How to implement interfaceless crawling using python + selenium + chromedriver

In the process of using selenium to crawl 12306, I found that phantomjs cannot be used to crawl, and chromedriver can be used. It should be that phantomjs is detected and banned by the website. Using chromedriver will display the interface again, and the crawling efficiency is low. Now I have two qu...

迷茫 2017-05-18 10:53:13

Python multi-threaded crawling files, how to set timeout and reconnection.

When using python to crawl data, enable multi-thread crawling in a single process. After all, I don’t have multiple processes because of intensive IO. The code is as follows {code...} However, as long as a thread's requests do not return a value, the thread will keep waiting and will not write, so t...

黄舟 2017-05-18 11:02:31

html - "DNS lookup failed" appears when crawling, but there is no problem when opening the web page. What is happening?

RT, when using python to crawl, everything was normal at the beginning, but suddenly the web page got stuck. After waiting for a while, it returned to normal. However, when crawling again at this time, it will report "DNS lookup failed". I tried many websites, but the web page failed. It o...

PHP中文网 2017-06-14 10:50:53

php dynamic web crawling

I'm trying the chrome-php/chrome library to read dynamically rendered web pages, but it's not returning anything. I am using php8.0 version. This is what I have implemented [I have followed the documentation] - useHeadlessChromium\BrowserFactory;$browserFactory=newBrowserFactory();//startsheadlessch...

P粉311089279 2023-09-01 18:42:58

php - snoopy crawler reports error 405 Not Allowed

Code $httpClass = new Snoopy();$httpClass->fetch('https://v.qq.com/');$url = $httpClass->results;print_r($url);die(); When crawling https://www.baidu.com/, it keeps reporting a 405 error. Crawling https://v.qq.com/ is a normal operation...

高洛峰 2017-05-16 12:58:40

python - Questions about crawling images?

Today's crawlers generally parse images from HTML, but the problem is that some image paths are written in JS. How to climb this? I know that you can view the pictures inside the source by opening the f12 point in the browser, but how do you use a crawler to crawl out the resources in the source?

我想大声告诉你 2017-06-28 09:25:40

java - Problem with crawling images?

I just modified the POM. You can download the source code again. This is the project address of the crawler I wrote. The project does not report an error, but the problem is that after downloading the image to the local area, the image is often incomplete, as follows: This is the core code for downl...

天蓬老师 2017-06-23 09:12:30

python - How to solve the memory leak and suspended animation of pyspider phantomjs?

When crawling, phantomjs will die. How to solve this problem. . .

黄舟 2017-05-18 10:59:23

javascript - Problem with nodejs crawling web pages

I plan to use nodejs to capture all the news on the website below. According to the general idea, first get the URL of each page of news, and then get the URL of each news and use request to fetch the content of each URL. That's it. But all the paging information of the following website, as well as...

阿神 2017-05-16 13:43:09

node.js - Questions about Node crawler crawling novels

{code...} The following is the code I requested. I crawled out the specific chapter list in another js and wrote it in json. Here I directly request each link: {code...} Just like this Same, after climbing for a while, it just stopped like this, as if it had crashed. No matter how long I waited, the...

巴扎黑 2017-05-16 13:38:49

Web crawler-Introduction to Python crawler

I am about to be a sophomore, I have studied Python by myself, and I know basic grammar. I want to learn crawling, but I feel that it involves a lot of knowledge. Is there anyone who has experienced it and can summarize what they know, or how to learn Python crawling?

欧阳克 2017-07-05 10:34:11

javascript - Jquery selector issue for crawling web pages

When using Jquery to analyze web pages, I feel that both writing methods should be able to choose. But in fact, the above writing method is OK, but the following URL is not: http://www.shindengen.co.jp/t... {code...} The dl I see in the F12 developer tool of chrome is <dl class="...

滿天的星座 2017-05-18 10:49:44

Upload a CSV file containing the URLs from the HTML page and use Flask to read the URLs you want to crawl

I currently need to make a web-based system that can upload a CSV file containing a list of URLs. After uploading, the system will read the URL line by line and will be used for the next step of crawling. Here, crawling requires logging into the website before crawling. I already have the source cod...

P粉799885311 2023-09-07 11:22:35

laravel - Crawling using Guzzle software, how to log in?

The author uses Guzzle in the laravel framework to crawl the data on my homepage, just for fun; but the page obtained is the login page; I have tried this way of writing, but it doesn't work {code...} I hope someone can tell me?

学习ing 2017-06-13 09:24:34

python - The number of Douyu followers is shown as a loading picture. How to crawl the number of followers?

Crawling the number of Douyu followers and displaying {code...} and displaying {code...} after the website is loaded. How should we crawl this kind of data?

巴扎黑 2017-07-05 10:34:08

A usage in python is unclear

In the process of writing a crawler in Python (crawling Wikipedia entries), the url["href"] appeared during the output process of using the iterator. I thought it should belong to a usage in the iterator, but I couldn't find it. Please help me with this usage. I mean, thank you for the url...

阿神 2017-05-18 10:59:46

python - Crawling WeChat public account articles requires entering verification code

The request is normal at first. If there are too many requests, this web page will be returned, prompting for verification. Now that a proxy is added and a header is added, this is returned. I wonder if there is any way to bypass this verification, or can the verification be simulated in the backgro...

过去多啦不再A梦 2017-06-28 09:22:11