SpeedAbout this page
Create a clear robots.txt policy and optional sitemap directive.
How to use it
- Enter the minimum information required for the intended output.
- Generate the result locally and read every option and warning.
- Check syntax, URLs, escaping and values against the relevant current specification.
- Test the output on one representative page or system before wider use.
How to interpret the result
Generated output from Generate a robots.txt file is a starting point that must match the real page, application and policy where it will be published.
Generation does not guarantee compatibility, search visibility, accessibility, security or compliance with third-party platform requirements.
Practical examples
Allow normal crawling
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Keep an internal area out of crawling
User-agent: * Disallow: /internal-search/ Disallow: /temporary/
Common mistakes to avoid
- Using Disallow as a privacy or access-control mechanism. Protect private content with authentication.
- Blocking CSS or JavaScript needed to render and understand public pages.
- Publishing “Disallow: /” on the production website after copying a staging configuration.
- Assuming robots.txt removes an already indexed URL. Use the appropriate indexing controls instead.
- Adding a sitemap URL that redirects, returns an error or contains non-canonical URLs.
Frequently asked questions
Where must robots.txt be published?
Publish it at the root of the exact protocol and hostname it controls, such as https://example.com/robots.txt.
Does robots.txt hide private information?
No. The file is public and crawling rules are not access control. Sensitive content needs authentication and appropriate permissions.
Can I include a sitemap?
Yes. Add an absolute Sitemap directive pointing to an accessible sitemap or sitemap index.
Should I block duplicate pages?
Usually canonicalisation, redirects and deliberate internal linking are better tools. Blocking can prevent a crawler from seeing a canonical signal.
Authoritative references
Reviewed: 21 August 2026 · SpeedVerge editorial team
Recommended next step
Validate the generated output, publish it through a reversible change and monitor the destination for errors.