Skip to main content

robots.txt Generator

Developer Tools

UA rules + AI bots + Sitemap

Quick Answer

robots.txt Generator: UA rules + AI bots + Sitemap. robots.txt Generator uses standard developer formulas to compute accurate results instantly.

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /tmp/

Sitemap: https://example.com/sitemap.xml

Syntax reference

User-agent: *Applies to all crawlers
Disallow: /admin/Block path prefix
Disallow:（留空）Empty = allow all
Allow: /public/Exception inside a disallow
Disallow: /*? Wildcard support
Disallow: /*.pdf$End anchor
Crawl-delay: 10Politeness delay
Sitemap: https://…Sitemap location
robots.txt is a public convention, not access control — malicious crawlers ignore it, and every line is visible to anyone at /robots.txt (never list secret paths, that just advertises them). For real protection use authentication. Also note: pages blocked by robots.txt can still appear in search results (without a snippet) if other sites link to them; use noindex meta for de-indexing. Google ignores Crawl-delay — manage its rate in Search Console.

Was this tool helpful?

How to Use

  1. Paste or type content: drop code/text into the robots.txt Generator or fill the parameters.
  2. Choose mode/options: switch format, rules or processing as needed.
  3. Live output: results appear instantly — nothing is submitted, data stays in your browser.
  4. One-click copy: copy the output straight into your development environment.

Key Points

  • robots.txt Generator is a free online developer tool — open and use, no sign-up.
  • All calculations run locally in your browser; your data never leaves the device.
  • Key capability: UA rules + AI bots + Sitemap.
  • Results come with formulas and reference tables so you can verify every number.

Formula & How It Works

robots.txt Generator formula

robots.txt Generator uses standard developer formulas to compute accurate results instantly.

FAQ

How do I use the robots.txt Generator?
Open the page and fill in the input fields — results update automatically. UA rules + AI bots + Sitemap.
Are the results accurate?
Yes. Standard formulas and conversion factors are used and verified.
Is it free? Do I need to sign up?
Completely free. No registration, login or download required.
Does it work on mobile?
Yes. The responsive design works on phones, tablets and desktops.
Is my data saved?
No. Everything runs locally in your browser.

Related Tools

Related Guides

Source & Updates

Based on RFC 4648, ECMA-262 and related specifications. Last updated: 2025-07

Last updated:

Formulas and content on this page are reviewed by the YiJianSuan team. For reference only — official sources prevail.

Maintained and periodically reviewed by the YiJianSuan editorial team — review process in ourEditorial Policy.

Cite this page: YiJianSuan. robots.txt Generator[EB/OL]. https://www.yijiansuan.com/en/c/dev/robots-txt

Results provided by this tool are for reference only and do not constitute professional advice.