Skip to content

Robots.txt Generator

Generate robots.txt with per-bot Allow and Disallow rules, crawl-delay, sitemap URLs and AI crawler blocks like GPTBot, or validate an existing file.

Robots.txt Generator

Presets

Host (optional)

Specifies the preferred domain (rarely used)

User-agent rules

seconds

Sitemaps

No sitemaps added

robots.txt output

5 lines
# robots.txt generated by AI Dev Hub
# https://aidevhub.io/robots-txt-generator/

User-agent: *
Allow: /

How it works

User-agent: Specifies which bot the rules apply to. Use * for all bots.

Allow/Disallow: Controls which paths the bot can or cannot crawl. Paths support wildcards (*).

Crawl-delay: Requests bots to wait X seconds between requests (not all bots honor this).

Sitemap: Tells search engines where to find your XML sitemap.

Common patterns

Disallow: /Block entire site
Disallow: /admin/Block /admin/ directory
Disallow: /*.pdf$Block all PDF files
Allow: /public/Allow /public/ directory

Updated . Provided as is. Check the output before you rely on it in production.

How to use Robots.txt Generator

  1. 1

    Specify crawlable pages

    Enter paths you want to allow search engines to crawl (/blog, /products). Or specify disallowed areas (/admin, /private).

  2. 2

    Add crawler rules

    Set rules per bot (Googlebot, Bingbot, GPTBot and others) with their own allow and disallow paths, and an optional crawl-delay.

  3. 3

    Define sitemaps

    Add URLs to your XML sitemaps. The generator creates proper Sitemap directives that search engines use to index efficiently.

  4. 4

    Test against URLs

    Switch to Validate and paste a robots.txt to check directive order, path syntax, crawl-delay values and sitemap URLs.

  5. 5

    Validate and deploy

    Validate the file, then download it and place it at yourdomain.com/robots.txt. Crawlers pick up the change the next time they fetch the file.

Questions and answers

What is robots.txt Generator & Validator?
robots.txt is a plain-text file at a site's root that tells crawlers which paths they may fetch, per user agent. This tool builds one from presets and per-bot Allow, Disallow and Crawl-delay rules with sitemap lines, can block AI crawlers such as GPTBot and CCBot, and checks pasted files for invalid directives.
Can robots.txt Generator block AI crawlers?
Yes. It includes options to block known AI scrapers like GPTBot, Google-Extended, CCBot, and others. This helps prevent your content from being used to train AI models without your consent.
For AI agents: how to call this tool

Machine-readable contract, endpoints and examples. Humans can ignore this section.

Best Path For Builders

Browser workflow

Runs instantly in the browser with private local processing and copy/export-ready output.

Browser Workflow

This tool is optimized for instant in-browser execution with local data handling. Run it here and copy/export the output directly.

/robots-txt-generator/

For automation planning, fetch the canonical contract at /api/tool/robots-txt-generator.json.