How to check whether a website is crawled by Baidu

How to check whether the website is crawled by Baidu: by analyzing the activity of Baidu spiders in the website log, crawling frequency, returned HTTP status code, etc. You can also view the log file in the root directory of the website, which records the visits and operations on the website.

The program used by Baidu to crawl web pages is called Baiduspider - Baidu Spider. We check the situation of the website being crawled by Baidu mainly for analysis. Baidu Spider in the website log Baiduspider activity: crawling frequency, returned HTTP status code.

How to view the log:

Through FTP, find a log file in the root directory of the website. The file name generally contains log. Download and decompress the notepad inside, which is It is the log of the website, recording the visits and operations of the website.

Because the conditions of each server and host are different, the log functions of different hosts record different contents, and some even have no log function.

The log content is as follows:

61.135.168.22 - - [11/Jan/2009:04:02:45 0800] "GET /bbs/thread-7303- 1-1.html HTTP/1.1" 200 8450 "-" "Baiduspider (http://www.baidu.com/search/spider.htm)"

Analysis:

GET / bbs/thread-7303-1-1.html represents, grab the page /bbs/thread-7303-1-1.html.

200 means successful capture.

8450 means 8450 bytes were captured.

If the format in your log is not like this, it means that the log format settings are different.

You can see 200 0 0 and 200 0 64 in many logs, which represent normal crawling.

The crawling frequency is obtained by checking the number of Baidu Spider crawls in the daily logs. There is no standardized timetable or frequency number for crawling frequency. We generally judge it by comparing logs over multiple days. Of course, we hope that Baidu Spider can crawl as many times as possible every day.

The above is the detailed content of How to check whether a website is crawled by Baidu. For more information, please follow other related articles on the PHP Chinese website!