Robots.txt Generator

Generate a deploy-ready robots.txt with search-engine and 2026 AI crawler rules. Pick a stack template, tune disallow paths, and copy the file. It is generated locally, and your site is never fetched or tested live.

Loading…
Robots.txt Generator
Configure crawl rules and AI agent policies, then copy a robots.txt block for your stack. Validation runs locally; FastMinify never requests your live site.

Developer notes

deployTarget changes the header comment and default disallow seeds until you edit paths manually. aiCrawlerPolicy adds named User-agent groups on top of the * group — it does not replace search-engine crawling rules on *. includeSitemap appends a Sitemap line only when the URL field is non-empty; use an absolute https:// URL in production. outputStyle minimal strips comment lines for paste into an existing file. This tool is separate from FastMinify production /robots.txt and from meta name="robots" on the Meta Tags Generator.

Options

  • AI crawler policy

    Adds named User-agent groups for 2026 AI crawlers. block-training is the default, and it blocks model-training bots while allowing search/browse agents.

  • Deploy target

    Changes the copy-target comment and default disallow path seeds for static root, Next.js public/robots.txt, or WordPress root.

  • Include sitemap line

    Appends Sitemap: {url} when the sitemap field is non-empty. Turn off for staging drafts without a production URL yet.

  • Output style

    commented adds section headers as # lines; minimal emits directives only for paste into an existing file.

When this is the right tool

Ship a static or JAMstack site with GPTBot/Google-Extended blocked but search/browse agents allowed. Drop public/robots.txt into a Next.js project with sensible /api/ and /_next/data/ disallows. Draft a WordPress root robots.txt before upload, without a live URL tester.

Related tools

Back to the SEO / Web Tools hub. Pair with Meta Tags Generator for meta robots vs robots.txt. Build a matching sitemap.xml and reference it with a Sitemap line. Add structured data with JSON-LD Schema Markup Generator. Decompose sitemap URLs with URL Parser, normalize path segments with Slugify, or format JSON-LD later with JSON Formatter.

Robots.txt Generator FAQ

Local crawl-file generator

Does FastMinify fetch or test my live robots.txt?

No. Generation and validation use only your form inputs. Nothing is crawled, proxied, or checked against your production domain.

Is this the same as FastMinify's own robots.txt?

No. FastMinify's /robots.txt serves this site. This generator builds a file for your project, with separate logic and separate output.

What do the AI crawler presets mean?

block-training disallows training crawlers (GPTBot, Google-Extended, CCBot, anthropic-ai). It still allows search and browse agents like ChatGPT-User and PerplexityBot. block-all-ai disallows every catalogued AI agent. custom lets you pick agents individually.

Can this emit meta robots tags?

No. Use the Meta Tags Generator sibling for <meta name="robots">. robots.txt and meta robots solve different layers of crawl control.

Where do I generate the sitemap.xml file?

Use the XML Sitemap Generator sibling. Paste your URL list, download sitemap.xml, then reference it here with Sitemap: https://example.com/sitemap.xml when include sitemap line is on.

SEO / Web Tools

Generate meta tags, Open Graph, Twitter Card, robots.txt, sitemap.xml and JSON-LD in your browser. Previews come from your input, and your site is never fetched.

Explore other tool categories