What is a Meta Robots Tag?
A meta robots tag is an HTML element used to instruct search engine crawlers on how to index and interact with a web page. It is placed within the <head> section of a webpage’s HTML and provides directives that control the page’s indexing, crawling, and presentation in search results.
Here’s an example of a meta robots tag:
<meta name="robots" content="noindex, nofollow">In this example:
noindextells search engines not to index the page.nofollowtells search engines not to follow any links on the page.
Why Are Meta Robots Tags Important?
Meta robots tags are crucial for controlling the visibility and indexing of web pages. They allow webmasters to:
- Prevent Indexing: Use the
noindexdirective to prevent a page from appearing in search results, which is useful for pages with thin content, admin sections, or thank you pages. - Control Link Following: The
nofollowattribute prevents search engines from following links on the page, which can help manage link equity and avoid passing SEO value to potentially untrusted sites. - Manage Snippets: Directives like
nosnippetprevent search engines from showing text snippets or previews of the page, which can help control what users see in search results. - Handle Cached Versions: The
noarchivedirective prevents search engines from showing cached versions of the page, which can be useful for content that frequently changes or for privacy reasons.
Meta Robots Tag Directives
Here are the most commonly used directives with the meta robots tag:
noindex: Instructs search engines not to index the page, preventing it from appearing in search results.nofollow: Tells search engines not to follow any links on the page.none: Equivalent to using bothnoindexandnofollow.all: Equivalent toindex, follow. It is generally unnecessary to use this directive as it is the default setting.noarchive: Prevents search engines from showing a cached version of the page in search results.nositelinkssearchbox: Prevents Google from showing a sitelinks search box for the page in search results.nosnippet: Instructs search engines not to display a text snippet or video preview from the page in search results.indexifembedded: Allows indexing of content embedded through iframes or similar HTML tags even if the main page has anoindexdirective.max-snippet: Specifies the maximum number of characters that Google can display in text snippets.max-image-preview: Controls the size of images that search engines can use in image snippets.max-video-preview: Sets the maximum number of seconds for video previews in search results.notranslate: Instructs Google not to offer translation for the web page.noimageindex: Prevents the indexing of images on the web page.unavailable_after: Specifies a date and time after which the page should not appear in search results.
How to Set Up the Meta Robots Tag
For websites built with CMS platforms like WordPress, setting up meta robots tags can be done through popular SEO plugins:
Using Yoast SEO Plugin:
- Go to the editing screen for the post or page.
- Scroll down to the Yoast SEO meta box.
- In the “Advanced” tab, use the dropdown labeled “Allow search engines to show this Post in search results?” to control the
noindexattribute.
Using RankMath Plugin:
- Edit the post or page.
- Open the “Advanced” tab in the RankMath meta box.
- Select the “No Index” option to apply the
noindexdirective.
For Non-CMS Websites:
- You can manually add the meta robots tag to the
<head>section of your HTML pages based on your needs.
Verification and Tools
To verify the meta robots tag implementation:
- Use tools like the Ahrefs SEO Toolbar to check the
indexabilitysection for both meta tags and HTTP headers. - Google Search Console can also provide insights into how Google views and interacts with your pages.
Summary
Meta robots tags are a powerful tool for controlling how search engines index and display your content. By correctly implementing and managing these tags, you can effectively guide search engine crawlers and enhance your site’s SEO and user experience.
