Robots.txt generator
Generate a robots.txt file with allow/disallow rules, crawl-delay and AI bot blocking. Preset templates for common CMS platforms.
- Free, no sign-up
- No data stored
- Reviewed 2026-01-15
Your details
Results
- Disallow rules
- 0
- Allow rules
- 0
- Size
- 262 B
- Ai bots blocked
- No
robots.txt
# robots.txt for https://example.com
# Generated by ayduke.com/seo/robots-txt-generator
User-agent: *
Allow: /
# Google advertising crawlers
User-agent: Mediapartners-Google
Allow: /
User-agent: AdsBot-Google
Allow: /
Sitemap: https://example.com/sitemap.xmlRules check
| Item | Value | Note |
|---|---|---|
| Check location | /robots.txt | Root only |
| Check case | robots.txt | Lowercase only |
| Check size | < 500 KB | Google limit |
| Check noindex | ✗ | Noindex unsupported |
Robots note
About the Robots.txt Generator
Build a robots.txt file with custom rules for different user-agents. Set paths to allow or disallow, add a crawl-delay, block AI training bots (GPTBot, CCBot, etc.) and keep Google AdsBot whitelisted for ad verification.
How to use the Robots.txt Generator
- Enter your domain for the Sitemap directive.
- Choose a preset or start from scratch.
- Add disallow/allow paths.
- Set crawl-delay if needed.
- Toggle AI bot blocking.
- Copy the output.
Frequently asked questions
Does robots.txt block indexing?
It blocks crawling, not indexing. A page can still appear in search results (with no snippet) if linked from elsewhere. Use a noindex meta tag to prevent indexing.
Should I block /admin/ paths?
Yes for crawler efficiency, but do not rely on robots.txt for security. Use authentication to protect admin areas.
What is AdsBot?
Google AdsBot checks landing page quality for ads. Blocking it can hurt your Quality Score. The "Allow AdsBot" option keeps it unblocked.