ToolAtlas
FavoritesRecent

robots.txt generator

Processed in this browser

Text limit: 2000000 characters

This tool never uploads your input.

Avoid shipping Disallow: / by accident. Google and Bing still need the file at https://example.com/robots.txt. We do not crawl you.

A marketing launch that accidentally blocked the whole origin. This page drafts robots.txt. You still place it at the site root. It is not Search Console.

Meta robots and OG tags are meta tag generator. Status codes are HTTP reference.

Draft at the site root

Disallow: / left on production is the usual accident. We draft Allow/Disallow. We do not fetch your live file, submit a sitemap, or override noindex. Hiding a URL in robots can keep Google from seeing a noindex tag. Place the file yourself at https://your-host/robots.txt.

How to use it

  1. List user-agents and paths you actually mean.
  2. Copy the text. Keep a Sitemap: URL only if that XML exists.
  3. Deploy and fetch https://your-host/robots.txt yourself.

Honest limits

Not a ranking lever we will claim. Not noindex. Not a crawl of your origin.

FAQ

Does this submit my sitemap?

No. A Sitemap: line is a hint. Search Console / Bing Webmaster are separate.

Will this override a noindex tag?

No. robots.txt can hide a URL from a crawler and also hide the noindex. Be careful with staging.

Google-specific directives?

We draft common Allow/Disallow. Exotic crawl-delay support varies by bot.

Can you audit my live file?

No fetch. Paste or write, then deploy.

Is my domain uploaded?

No.

Related tools