How to use
- Choose Allow all, Block all, WordPress or Custom under Preset.
- Enter your rules in Disallow paths and Allow paths, one path per line.
- Add your sitemap addresses to Sitemap URL(s), one full http(s) URL per line.
- Enter a URL or path in Test a URL and choose a User-agent to check which rule matches.
- Click Download to save robots.txt for upload to your site root.
Examples
WordPress site with a sitemap
Preset: WordPress Disallow paths: /wp-admin/ Allow paths: /wp-admin/admin-ajax.php Sitemap URL(s): https://example.com/sitemap.xml Include header comment: off Test a URL: /wp-admin/admin-ajax.php User-agent: Googlebot
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap.xml
Block AI training crawlers
Preset: Allow all Block AI training Sitemap URL(s): https://example.com/sitemap.xml Include header comment: on Test a URL: /blog/summer-gardening User-agent: GPTBot
# robots.txt generated by AllRight Tools – https://allrighttools.com/robots-txt-generator/ User-agent: * Allow: / User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: Meta-ExternalAgent User-agent: Amazonbot User-agent: cohere-ai Disallow: / Sitemap: https://example.com/sitemap.xml
FAQ
Can robots.txt protect private pages?
Robots.txt is a request to crawlers, not access control. Do not rely on it to protect private content; some user-triggered fetchers may not apply it.
Can I check a website's live robots.txt?
The tester checks only the generated file or robots.txt text you paste. Entering a URL tests its path and query against that text; it does not fetch the website or its live robots.txt.
How do wildcards work in robots.txt?
An asterisk (*) matches any sequence of characters. A dollar sign ($) at the end anchors the rule to the end of the URL path, so a URL with a query string, such as /file.pdf?x=1, does not match /*.pdf$. The longest matching rule wins, Allow wins a tie, and path matching is case-sensitive.
Are my robots.txt rules saved between visits?
Paths, sitemap URLs, custom user-agents and tester input are not saved. Only your Preset, Crawl-delay, Include header comment setting and the settings of the built-in bots are saved in your browser's localStorage.