python - pyspider scheduled crawling problem

When writing the crawler, I found that after setting every in the code, after crawling once on the 21st, I saw that the result was not updated today, and the lastcrawltime was still on the 21st. Is my parameter setting incorrect?

我想大声告诉你 2017-05-18 10:53:29 0  2  979

python - scrapy crawls many more pages than actually enters items?

{Code...} Why do I visit more pages than actually enter items when I use scrapy to crawl? Is there any way to solve the problem that after crawling for a long time, there are not many pieces of data in the items?

PHP中文网 2017-05-18 10:47:40 0  1  672

javascript - Why can't I crawl the page if I use phantomjs to crawl it if there is a lot of data on the page?

I want to crawl a Taobao search page. It is a page generated by js rendering, so I choose to crawl it with phantomjs. But a problem occurred during the fetching process. When there are more than two search results, no data can be obtained. But the screenshot shows that the data is loading normally. ...

習慣沉默 2017-07-05 10:50:52 0  1  913

How to crawl js processed code in java

Page address: http://acm.hdu.edu.cn/showpro... Goal of crawling: If you want to crawl the code of these formulas, the code you see when pressing F12 on Chrome: But the code crawled is as follows: This code does not display the correct formula. It seems that these codes are all generated by js. How t...

为情所困 2017-05-17 10:04:18 0  2  643

python - The number of Douyu followers is shown as a loading picture. How to crawl the number of followers?

Crawling the number of Douyu followers and displaying {code...} and displaying {code...} after the website is loaded. How should we crawl this kind of data?

巴扎黑 2017-07-05 10:34:08 0  1  993

How to prevent malicious ddos ​​crawling on nginx

First of all, I have no objection to others crawling the content of my website. I don’t necessarily strictly limit other people’s crawling. However, some people’s crawling has no bottom line at all. They use one script or even multiple scripts to crawl a certain website concurrently. The content of ...

黄舟 2017-05-16 17:30:17 0  4  1080

Curl crawls web pages for data

If the URL link is https://mbd.baidu.com/newspage/data/landingsuper?context=%7B%22nid%22%3A%22news_4480296238548479181%22%7D&n_type=0&p_from=1, how to crawl the web page information?

phpcn_u68041 2017-12-07 10:08:47 0  5  1221

How does the PHP source code crawl data by customizing the interface to obtain the proxy IP?

How does the PHP source code crawl data by customizing the interface to obtain the proxy IP? How does the PHP source code crawl data by customizing the interface to obtain the proxy IP?

chen 2022-11-04 23:36:05 0  0  580

python - Questions about crawling images?

Today's crawlers generally parse images from HTML, but the problem is that some image paths are written in JS. How to climb this? I know that you can view the pictures inside the source by opening the f12 point in the browser, but how do you use a crawler to crawl out the resources in the source?

我想大声告诉你 2017-06-28 09:25:40 0  2  1044

php crawls web page data

I have some needs such as crawling the Baidu takeout merchant's webpage https://wmpass.baidu.com/ucen..., but this webpage is written using a modular front-end framework, and I can only see some when viewing the source code. The source code before rendering, the querylist I used to get is not after ...

PHP中文网 2017-05-24 11:34:27 0  1  889

Python multi-threaded crawling files, how to set timeout and reconnection.

When using python to crawl data, enable multi-thread crawling in a single process. After all, I don’t have multiple processes because of intensive IO. The code is as follows {code...} However, as long as a thread's requests do not return a value, the thread will keep waiting and will not write, so t...

黄舟 2017-05-18 11:02:31 0  1  839

node.js - node crawler, how to use IP pool to prevent anti-crawling?

The problem is this. I am a newbie who just started learning node. Of course, it is obviously for crawlers. Then I was reading a novel recently, but there were too many ads on those free novel websites, so I planned to write a crawler to crawl the entire novel. However, the number of url requests wa...

为情所困 2017-06-05 11:13:33 0  1  1238

Dear python crawler experts, take a look and see how to deal with anti-crawling on this website.

https://www.everysaving.co.uk crawls the data of this website through python, but the data cannot be returned! I added the header and proxy IP to crawl, but it didn't work. I hope you guys can give it a try. . .

曾经蜡笔没有小新 2017-05-18 11:01:00 0  4  776

javascript - node crawls Weibo

I was new to node and wanted to write a crawler to crawl Sina Weibo comments, but I found that the page was dynamically generated by JS and could not be crawled using the http module, so I used phantomjs to crawl it (I heard it would be slower, and it took nearly a month to run) 15 minutes, this is ...

仅有的幸福 2017-06-30 10:00:08 0  1  1001

javascript - Redirection problem when superagent crawls website

I used superagent to crawl the web page and encountered a redirection. I couldn't get the response body after the redirection. How to solve it? How to get the redirected webpage. I wanted to get the res of the 501 page, but it jumped to the 37018 page, causing me to get Empty res to 501 page. I test...

迷茫 2017-05-31 10:38:19 0  1  858

HTML elements cannot be located during web crawling. This element is visible in the site inspection tool

I'm trying to crawl the titles of all tables from this URL: https://www.nature.com/articles/s41586-023-06192-4 I can find this HTML element on the website: <bid="Tab1"data- test="table-caption">Table1CalculatedCt–M–Ctangles</b>I cannot crawl this title

P粉225961749 2023-09-19 12:41:46 0  1  602

python - Use scrapy framework to crawl Baidu pictures and get blocked

The request address URL is the address of json obtained through firefox. It can be opened with a browser, but it was banned when crawling with scrapy. Please solve it. https://image.baidu.com/searc...

给我你的怀抱 2017-05-24 11:34:48 0  3  672

node.js - Questions about Node crawler crawling novels

{code...} The following is the code I requested. I crawled out the specific chapter list in another js and wrote it in json. Here I directly request each link: {code...} Just like this Same, after climbing for a while, it just stopped like this, as if it had crashed. No matter how long I waited, the...

巴扎黑 2017-05-16 13:38:49 0  1  623

Python crawls web pages with garbled requests

**I crawled data from referee documents before. After re-running the crawler during this period, I found that the web page data could not be obtained. After searching, I found that the source code of the requests web page returned garbled characters** (Part of the returned data is intercepted as fol...

phpcn_u1582 2017-06-22 11:51:56 0  4  1262

laravel - Crawling using Guzzle software, how to log in?

The author uses Guzzle in the laravel framework to crawl the data on my homepage, just for fun; but the page obtained is the login page; I have tried this way of writing, but it doesn't work {code...} I hope someone can tell me?

学习ing 2017-06-13 09:24:34 0  1  827