Robots.txt Generator
Generate a deploy-ready robots.txt with search-engine and 2026 AI crawler rules. Pick a stack template, tune disallow paths, and copy the file — generated locally, your site is never fetched or tested live.
Developer notes
• deployTarget changes the header comment and default disallow seeds until you edit paths manually. • aiCrawlerPolicy adds named User-agent groups on top of the * group — it does not replace search-engine crawling rules on *. • includeSitemap appends a Sitemap line only when the URL field is non-empty; use an absolute https:// URL in production. • outputStyle minimal strips comment lines for paste into an existing file. • This tool is separate from FastMinify production /robots.txt and from meta name="robots" on the Meta Tags Generator.
Options
AI crawler policy
Adds named User-agent groups for 2026 AI crawlers. block-training is the default — blocks model-training bots while allowing search/browse agents.
Deploy target
Changes the copy-target comment and default disallow path seeds for static root, Next.js public/robots.txt, or WordPress root.
Include sitemap line
Appends Sitemap: {url} when the sitemap field is non-empty. Turn off for staging drafts without a production URL yet.
Output style
commented adds section headers as # lines; minimal emits directives only for paste into an existing file.
When this is the right tool
• Ship a static or JAMstack site with GPTBot/Google-Extended blocked but search/browse agents allowed. • Drop public/robots.txt into a Next.js project with sensible /api/ and /_next/data/ disallows. • Draft a WordPress root robots.txt before upload — without a live URL tester.
Related tools
Hub: SEO / Web Tools. Pair with Meta Tags Generator for meta robots vs robots.txt. Build matching sitemap.xml and reference it with the Sitemap: line. Add structured data with JSON-LD Schema Markup Generator. Decompose sitemap URLs with URL Parser, normalize path segments with Slugify, or format JSON-LD later with JSON Formatter.
Robots.txt Generator FAQ
Local crawl-file generator
Does FastMinify fetch or test my live robots.txt?
No. Generation and validation use only your form inputs. Nothing is crawled, proxied, or checked against your production domain.
Is this the same as FastMinify's own robots.txt?
No. FastMinify's /robots.txt serves this site. This generator builds a file for your project — separate logic, separate output.
What do the AI crawler presets mean?
block-training disallows training crawlers (GPTBot, Google-Extended, CCBot, anthropic-ai) while allowing search/browse agents like ChatGPT-User and PerplexityBot. block-all-ai disallows every catalogued AI agent. custom lets you pick agents individually.
Can this emit meta robots tags?
No — use the Meta Tags Generator sibling for <meta name="robots">. robots.txt and meta robots solve different layers of crawl control.
Where do I generate the sitemap.xml file?
Use the XML Sitemap Generator sibling — paste your URL list, download sitemap.xml, then reference it here with Sitemap: https://example.com/sitemap.xml when include sitemap line is on.
SEO / Web Tools
Generate meta tags, Open Graph, and Twitter Card output locally — previews simulated from your inputs, never fetched from your live site.
Explore other tool categories
Minify
Shrink code and assets for production — minify JavaScript, CSS, HTML, JSON and XML before gzip or CDN deploy.
Unminify
Expand minified or compressed code — unminify JavaScript, CSS, TypeScript, SCSS, LESS, SQL, YAML and other formats when debugging or reviewing.
Conversion
Transform data between JSON, YAML, XML and CSV locally — no server uploads.
Encoding
Encode or decode Base64, URL components and HTML entities — client-side only.
Serialization
Serialize and deserialize PHP data structures beside JSON workflows.
JSON Tools
Validate, format, diff and explore JSON payloads — complementary to minifiers and converters.
SVG Tools
Preview, optimize with SVGO, export Data URI, resize, beautify, convert to JSX and validate SVG — all in your browser.
Beautify
Make code readable with consistent indentation — beautify JavaScript, CSS, TypeScript, SCSS, LESS, Markdown, GraphQL, SQL, YAML and more in your browser.
CSS preprocessors
Compile SCSS or LESS to standard CSS in the browser — pair with beautify or minify for a full stylesheet workflow.
Color & CSS Tools
Paste-local WCAG contrast checking with modern CSS color syntax — pair and matrix views, never uploaded.
DevOps & Infra
Terraform HCL format/validate/minify, Dockerfile format & lint, Docker Compose and .env validation.
CI/CD
GitHub Actions and GitLab CI — format YAML and check workflow/job structure in your browser.
Logs & observability
Line-oriented JSON/NDJSON and Nginx/Apache access logs — format, filter, validate, and CSV/TSV.
Kubernetes
Multi-doc manifests, structural validate, Ingress/Deployment starters, and Helm values formatting.
Networking / IP
CIDR, subnets, and IPv4/IPv6 helpers — in your browser.
Security & tokens
JWKS, X.509 certificates, and secrets scan — browser-local.
API & schemas
OpenAPI tools to format, validate, and lint your specs.
HTTP Tools
Paste-local cURL, HTTP messages, URLs, HAR files, and Cookie / Set-Cookie headers — format and inspect in your browser, never sent.
Developer utilities
Timestamps, UUID, ULID, Nanoid, cron, passwords, regex, slugify, number bases, case, text diff, and chmod — all client-side.
Text & Markdown
Count words, build Markdown TOCs, and clean line lists — all in your browser.
AI & LLM
Token counting, pricing estimates, and context-window fit — 100% browser-local.
Test Data
Synthetic field values for fixtures and DB seed — reserved-range phone numbers first.