SEO Glossary /

X-Robots-Tag

The X-Robots-Tag is an HTTP header used to communicate instructions to web crawlers and search engine bots regarding how they should handle the indexing and serving of a web page or resource. It allows website owners to control the indexing and appearance of their pages in search results on a page-by-page or resource-by-resource basis.

The X-Robots-Tag header can include one or more directives, separated by commas. Some common directives are:

  • noindex: Instructs search engines not to index the page or resource.
  • nofollow: Instructs search engines not to follow the links on the page.
  • noarchive: Instructs search engines not to cache or display a cached version of the page.
  • nosnippet: Instructs search engines not to display a snippet or description of the page in search results.
  • max-snippet:[number]: Specifies the maximum character length for snippets in search results.
  • max-image-preview:[setting]: Specifies the maximum size of image previews in search results (none, standard, large).
  • max-video-preview:[number]: Specifies the maximum number of seconds for video previews in search results.

Example usage of X-Robots-Tag:

X-Robots-Tag: noindex, nofollow, max-snippet:150, max-image-preview:standard

In this example, the directives instruct search engines not to index the page, not to follow its links, to limit the snippet length to 150 characters, and to show standard-sized image previews in search results.

The X-Robots-Tag header provides more granular control over indexing and serving than the robots meta tag, which is placed within the HTML section of a page.

While the robots meta tag is limited to controlling indexing and link following, the X-Robots-Tag header offers additional directives for managing snippets, previews, and caching.

Using the X-Robots-Tag header, website owners can fine-tune how their pages appear in search results and control the information that search engines display to users. This can be useful for managing duplicate content, preventing the indexing of sensitive pages, or customizing the appearance of search listings to optimize click-through rates and user engagement.