What is robots.txt?

Robots.txt is the first file that search engines look at when they visit a website. It is a text file used to specify the scope of crawling of website content by search engines. When a search spider visits a site, it will first check whether robots.txt exists in the root directory of the site. If it exists, it will determine the scope of the visit based on the content in the file.

In the process of website construction, we will have some content that we do not want to be crawled by search engines or do not want it to appear on the Internet, so what should we do? ? How do I tell search engines not to crawl my xx content? This is where robots come in handy.

Robots.txt is the first file that search engines look at when visiting a website. The Robots.txt file tells the spider what files on the server can be viewed.

When a search spider visits a site, it will first check whether robots.txt exists in the root directory of the site. If it exists, the search robot will determine the scope of access based on the contents of the file; if If the file does not exist, all search spiders will be able to access all pages on the website that are not password protected.

Syntax: The simplest robots.txt file uses two rules:

• User-Agent: The robot to which the following rules apply

• Disallow: The web page to be blocked

But we need to pay attention to a few points:

1.robots.txt must be stored in the root directory of the website,

2. Its naming Must be robots.txt, and the file name must be all lowercase.

3.Robots.txt is the first page that search engines visit the website

4.Robots.txt must specify user-agent

robots.txt Misunderstandings

Misunderstanding 1: All files on my website need to be crawled by spiders, so there is no need for me to add the robots.txt file. Anyway, if the file does not exist, all search spiders will be able to access all pages on the website that are not password protected by default.

Whenever a user attempts to access a URL that does not exist, the server will record a 404 error (file cannot be found) in the log. Whenever a search spider looks for a robots.txt file that does not exist, the server will also record a 404 error in the log, so you should add a robots.txt to your website.

Misunderstanding 2: Setting all files in the robots.txt file to be crawled by search spiders can increase the inclusion rate of the website.

Even if the program scripts, style sheets and other files in the website are included by spiders, it will not increase the website's inclusion rate and will only waste server resources. Therefore, you must set it in the robots.txt file not to allow search spiders to index these files.

Specific files that need to be excluded are detailed in the article Tips on Using Robots.txt.

Misunderstanding 3: Search spiders waste server resources when crawling web pages. All search spiders set in the robots.txt file cannot crawl all web pages.

If this is the case, the entire website will not be indexed by search engines.

robots.txt usage tips

1. Whenever a user tries to access a URL that does not exist, the server will record a 404 error (File cannot be found) in the log ). Whenever a search spider looks for a robots.txt file that doesn't exist, the server will also record a 404 error in the log, so you should add a robots.txt to your site.

2. Website administrators must keep spider programs away from certain directories on the server - to ensure server performance. For example: most website servers have programs stored in the "cgi-bin" directory, so it is a good idea to add "Disallow: /cgi-bin" to the robots.txt file to prevent all program files from being indexed by spiders. Can save server resources. Files that do not need to be crawled by spiders in general websites include: background management files, program scripts, attachments, database files, encoding files, style sheet files, template files, navigation pictures and background pictures, etc.

The following is the robots.txt file in VeryCMS:

User-agent: *

Disallow: /admin/ Background management file

Disallow: / require/ Program file

Disallow: /attachment/ Attachment

Disallow: /images/ Picture

Disallow: /data/ Database file

Disallow: / template/ template file

Disallow: /css/ style sheet file

Disallow: /lang/ encoding file

Disallow: /script/ script file

3. If your website has dynamic web pages, and you create static copies of these dynamic web pages to make them easier for search spiders to crawl. Then you need to set up settings in the robots.txt file to prevent dynamic web pages from being indexed by spiders to ensure that these web pages will not be regarded as containing duplicate content.

4. The robots.txt file can also directly include links to the sitemap file. Like this:

Sitemap: http://www.***.com/sitemap.xml

The search engine companies that currently support this include Google, Yahoo, Ask and MSN. Chinese search engine companies are obviously not in this circle. The advantage of this is that the webmaster does not need to go to the webmaster tools or similar webmaster sections of each search engine to submit his own sitemap file. The search engine spider will crawl the robots.txt file and read the content in it. sitemap path, and then crawl the linked web pages.

5. Proper use of the robots.txt file can also avoid errors during access. For example, you can’t let searchers go directly to the shopping cart page. Since there is no reason for the shopping cart to be included, you can set it in the robots.txt file to prevent searchers from entering the shopping cart page directly

The above is the detailed content of What is robots.txt?. For more information, please follow other related articles on the PHP Chinese website!