Robots Meta tags: one of the most common mistakes in SEO
As one of the 50+ elements used for SEO evaluation, SEO MASTER Express checks the description for the robots.txt file, commonly known as the “Robots Meta tag”.
What is the “Robots Meta tag”?
The Robots Meta tag allows you to control the behavior of search engine crawlers by adding a tag such as “noindex”, “nofollow”, or “noarchive” to a specific page.
Robots Meta tags are usually added to a page that is not necessary to display (or you don’t want it to be shown) in the search results page, for example a “support contact” form.
The benefits of setting a Robots Meta tag
Crawlers automatically prioritize the order in which they crawl web pages. However in cases where a website has a large number of pages, a crawler may still need to crawl unnecessary pages, resulting in it taking a longer time for the crawler to reach the important pages that should be crawled and shown in search results.
In this situation, you can add a Robots Meta tag that directs a crawler to avoid unnecessary pages, in order to increase the efficiency of the crawling process within the website.
Uploading a robots.txt file to your website’s top directory on the FTP server and registering in the webmaster tools of a search engine will provide the same effect as adding a Robots Meta tag.
If “noindex” or “nofollow” tags are added to a page that you want displayed in the search results page, they should be removed.
Examples of Robots Meta tags
“noindex” is used to prevent a page from being indexed.
<meta name=”robots” content=”noindex”>
“noarchive” is used to prevent a page from being stored in the search engine’s database.
<meta name=”robots” content=”noarchive”>
“none” is used to prevent a page from being indexed or being crawled to the destination link.
<meta name=”robots” content=”none”>
Common mistakes in writing a Robots Meta tag
If you intended to set the Robots Meta tag to block a particular page but mistakenly blocked an entire directory, all the pages immediately below that directory would no longer appear in the search results.
This mistake often happens with CMS, because any fixes in the header will affect all of the pages. In this situation, you will need to write a PHP to set a condition, or you will need to manage the crawler by setting robots.txt with no description.
Check your Robots Meta tag
SEO MASTER Express identifies errors and provides advice when Robots Meta tags are set within a page.
*If you have intentionally set a Robots Meta tag to control crawlers so that they only crawl a particular file or directory, that generally doesn’t need correcting.
Download 14 Days Free Trial Now!