Guides
X-Robots-Tag: The Complete Guide (with Examples)
On this page
The X-Robots-Tag is an HTTP response header that tells search engines how to index a resource — it does everything the <meta name="robots"> tag does, but because it lives in the HTTP header instead of the HTML, it works on any file type, including PDFs, images, and other non-HTML files. If you’ve ever needed to noindex a PDF (which has no <head> to put a meta tag in), the X-Robots-Tag is the answer. Here’s how it works and when to reach for it.
What is the X-Robots-Tag?
Both the meta robots tag and the X-Robots-Tag deliver the same indexing directives — noindex, nofollow, noarchive, nosnippet, and so on. The difference is where they live:
- Meta robots tag — an HTML tag in the page
<head>. Only works for HTML pages. - X-Robots-Tag — an HTTP header sent with the server response. Works for HTML and every other file type.
That’s the whole point: the X-Robots-Tag lets you control indexing for resources that can’t hold a meta tag — PDFs, images, videos, spreadsheets, plain-text files.
When should you use the X-Robots-Tag?
Reach for it when a meta tag won’t work or isn’t practical:
- Non-HTML files.
noindexa PDF, image, or document that’s ranking when it shouldn’t. This is the classic use case. - At scale, by pattern. Apply a directive to many files at once via server config — e.g.
noindexevery file in/private/or every.pdf. Far easier than editing each file. - Dynamically. Set the header conditionally in server logic (by URL pattern, file type, or query) without touching page templates.
- When you can’t edit the HTML but you can configure the server.
X-Robots-Tag examples
Apache (in .htaccess or server config) — noindex all PDFs:
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>Nginx — noindex a location:
location /private/ {
add_header X-Robots-Tag "noindex, nofollow";
}As a raw HTTP response header:
X-Robots-Tag: noindexYou can also target a specific crawler: X-Robots-Tag: googlebot: noindex applies only to Googlebot. And you can combine directives: X-Robots-Tag: noindex, nofollow, noarchive.
The one rule that trips everyone up
For any noindex to work — meta tag or X-Robots-Tag — Google must be able to crawl the resource. The directive is delivered when Google fetches the file. If you also block the URL in robots.txt, Google never fetches it, never sees the header, and the page can stay indexed. This is the exact trap behind the Indexed, though blocked by robots.txt warning. So: to remove something, use noindex (via header or meta) and keep it crawlable — don’t Disallow it in robots.txt at the same time. The full crawl-vs-index distinction is in noindex vs robots.txt disallow.
How to check the X-Robots-Tag
Because it’s in the HTTP header, it’s invisible in the page source. To see it:
- Browser DevTools → Network tab → click the request → Response Headers.
curl -I https://example.com/file.pdfshows the response headers.- Search Console URL Inspection reports the effective indexing directive Google sees.
This invisibility is also why a mystery noindex is sometimes an X-Robots-Tag header nobody remembered setting — always check headers, not just HTML, when a page won’t index.
The bottom line
The X-Robots-Tag is the HTTP-header way to control indexing, and its superpower is working on non-HTML files like PDFs and images, plus applying directives at scale via server config. Use it wherever a meta robots tag can’t reach — just remember the golden rule: the resource must stay crawlable for the noindex to be seen. Keep it out of robots.txt, and check response headers when a page’s indexing behaves mysteriously.
If clear technical-SEO answers are useful to you, I send them occasionally — subscribe below, and grab the free GEO Starter Checklist.
Frequently asked questions
What is the X-Robots-Tag?
It is an HTTP response header that gives search engines indexing directives like noindex and nofollow. It does the same job as the meta robots tag, but because it lives in the HTTP header rather than the HTML, it works on any file type, including PDFs, images and other non-HTML resources.
When should I use X-Robots-Tag instead of a meta robots tag?
Use it for non-HTML files that cannot hold a meta tag (like PDFs and images), when you want to apply a directive to many files at once via server config, when you need to set it dynamically, or when you can configure the server but cannot edit the HTML.
How do I add an X-Robots-Tag?
Set it in your server configuration. On Apache, use a Header directive (often in .htaccess) matched to file types; on Nginx, use add_header in a location block. For example, X-Robots-Tag: noindex, nofollow applied to all .pdf files noindexes every PDF at once.
Why is my X-Robots-Tag noindex not working?
The most common reason is that the resource is also blocked in robots.txt, so Google never crawls it and never sees the header — leaving it potentially still indexed. For a noindex to work, the resource must remain crawlable. Remove the robots.txt disallow and let Google fetch the file.