HTMLRobots.txt Generator
Generate a robots.txt with sensible presets, custom rules, a sitemap line, and a copyable crawler policy pack that pairs robots.txt with llms.txt and a starter deployment check.
Ctrl+Enter to run | Ctrl+K to switch tools | Recent button shows input history
Repeating this work? Turn this kind of browser utility into an API batch, MCP prompt, extension workflow, or shared team recipe.
Buy Pro Browse tool packsFAQ
Can robots.txt block every AI system?
No. robots.txt is a public crawl directive that cooperative crawlers may honor. It is not access control. Use authentication or server rules for private content.
What is in the crawler policy pack?
The pack includes the generated robots.txt, publication paths, llms.txt and sitemap reminders, review notes, and a starter Node check for the live files.
Should I add an llms.txt file too?
Use robots.txt for crawl directives and llms.txt as a public Markdown guide to important content. The llms.txt convention is proposed and useful for discovery, not a guaranteed ranking signal.