SEO (Search Engine Optimization)
Robots Meta Directives
Robots Meta Directives are HTML meta tags that are used to control how search engine robots crawl and index web pages. These directives, placed in the header section of a web page, can indicate to search engine crawlers whether to index or not to index the content of a page, and whether to follow or not to follow the links on a page. The two most common robots meta directives are:
HTML Meta Directives
| Directive | Purpose |
|---|---|
| "index" | Allow page to be indexed by search engines |
| "noindex" | Prevent page from being indexed by search engines |
| "follow" | Allow search engines to follow links on the page |
| "nofollow" | Prevent search engines from following links on the page |
| "archive" | Allow search engines to store a cached copy of the page |
| "noarchive" | Prevent search engines from storing a cached copy of the page |
| "snippet" | Allow a description or snippet of the page to be displayed in search results |
| "nosnippet" | Prevent a description or snippet of the page from being displayed in search results |
| "odp" | Allow the Open Directory Project (ODP) description of the page to be used in search results |
| "noodp" | Prevent the Open Directory Project (ODP) description from being used in search results |
| "imageindex" | Allow search engines to index images on the page |
| "noimageindex" | Prevent search engines from indexing images on the page |
| "noydir" | Instruct Yahoo! not to use the Yahoo! Directory description of the page |
| "unavailable_after" | Specify a date and time after which the page should not be crawled or indexe |
Here are some examples of how the Robots Meta Directives can be used in a HTML code:
- To allow a page to be indexed:
<meta name="robots" content="index">
- To prevent a page from being indexed:
<meta name="robots" content="noindex">
- To allow search engines to follow links on the page:
<meta name="robots" content="follow">
- To prevent search engines from following links on the page:
<meta name="robots" content="nofollow">
- To allow search engines to store a cached copy of the page:
<meta name="robots" content="archive">
- To prevent search engines from storing a cached copy of the page:
<meta name="robots" content="noarchive">
- To allow a description or snippet of the page to be displayed in search results:
<meta name="robots" content="snippet">
- To prevent a description or snippet of the page from being displayed in search results:
<meta name="robots" content="nosnippet">
- To allow the Open Directory Project (ODP) description of the page to be used in search results:
<meta name="robots" content="odp">
- To prevent the Open Directory Project (ODP) description from being used in search results:
<meta name="robots" content="noodp">
- To allow search engines to index images on the page:
<meta name="robots" content="imageindex">
- To prevent search engines from indexing images on the page:
<meta name="robots" content="noimageindex">
- To instruct Yahoo! not to use the Yahoo! Directory description of the page:
<meta name="robots" content="noydir">
- To specify a date and time after which the page should not be crawled or indexed:
<meta name="robots" content="unavailable_after: Wed, 15 May 2021 12:00:00 GMT">
Mixing HTML meta directives
When using HTML meta directives, it's possible to mix multiple directives on a single page to achieve different results. For example, you can use both the "robots" meta tag and the "description" meta tag on a page to control the behavior of search engine robots and provide a brief summary of the page's contents, respectively.
Here's an example of mixing the "robots" and "description" meta tags:
<html>
<head>
<meta name="robots" content="noindex, follow">
<meta name="description" content="This is a brief summary of my page's contents">
</head>
<body>
<!-- Page contents go here -->
</body>
</html>
In this example, the "robots" meta tag tells search engine robots not to index the page, but to follow the links on the page. The "description" meta tag provides a brief summary of the page's contents, which may be displayed in search engine results.
By mixing different HTML meta directives, you can fine-tune the behavior of search engines when crawling and indexing your website. It's important to be cautious and careful when using these directives, as they can have significant impact on the visibility of your website in search results.
Targeting HTML meta tags
<meta name="googlebot" content="noindex" />
<meta name="slurp" content="noindex" />
The above HTML code contains two meta tags specifying directives for search engine crawlers.
The first meta tag is for "googlebot", which is the crawler used by Google's search engine. The "content" attribute is set to "noindex", which means that Google's search engine should not index the content of the page. This means that the page will not appear in search results.
The second meta tag is for "slurp", which is the crawler used by Yahoo! search engine. The "content" attribute is also set to "noindex", which means that Yahoo! search engine should not index the content of the page.
By adding these meta tags to your page, you can prevent search engines from indexing the content of the page, making it not visible in search results. However, it's important to note that not all search engines recognize these meta tags, and some may ignore them.