How to prevent robots from crawling php

How to prohibit robots from crawling php: 1. Write "Disallow: /*?*" in the robots.txt file; 2. Add the rule "User-agent:* Allow" to the robots.txt file : .html$ Disallow: /".

The operating environment of this article: windows7 system, PHP7.1 version, DELL G3 computer

Robots prohibit search engines from crawling PHP dynamics URL

The so-called dynamic URL means that the URL includes? , & and other character-type URLs, such as: news.php?lang=cn&class=1&id=2. After we turn on the pseudo-static function of the website, it is necessary to avoid searching for the SEO of the website. The engine crawls the dynamic URL of our website.

Why do this? This is because the search engine will trigger some actions on the website after crawling the same page twice but it is finally determined to be the same page. It is not clear how to punish it specifically. In short, it is not conducive to the SEO of the entire website. of. So how to prevent search engines from crawling the dynamic URL of our website?

Using the robots.txt file can solve this problem. Please see the detailed operations below.

We know that dynamic pages all have a common feature, that is, there will be a "?" question mark symbol in the link, so we can write the following rules in the robots.txt file:

User-agent: *
Disallow: /*?*

This will prevent search engines from crawling the dynamic links of the entire website. In addition, if we only want search engines to crawl specified types of files, such as static pages in html format, we can add the following rules to robots.txt:

User-agent: *
Allow: .html$
Disallow: /

Also remember to add the written robots. txt file in the root directory of your website, otherwise it will not work. In addition, there is a simple shortcut for writing rules. Log in to Google Webmaster Tools, just write the rules in it, and then generate the robots.txt file.

[Recommended learning: "PHP Video Tutorial"]

The above is the detailed content of How to prevent robots from crawling php. For more information, please follow other related articles on the PHP Chinese website!