Check
Do not use robots.txt to protect private content. Anyone can read the file, and blocked URLs can still be discovered from links.
Technical SEO tool
Robots.txt Generator
Generate a simple robots.txt file for public sites, publisher tools, or staging environments. Add a sitemap reference, review crawl-risk warnings, and copy a clean file before deploying it.
Generate robots.txt
Choose a preset, edit crawl rules, and copy a robots.txt file with an optional sitemap reference. Use the warnings before publishing.
Allow public pages and keep admin, API, account, and checkout paths out of crawl queues.
One path per line. Paths are relative to the site root.
Use directory slashes carefully, such as /admin/.
The generator converts a relative path into an absolute Sitemap line.
Review before publishing
How To Use This Robots.txt Generator
Start with the production domain and choose a preset. Most public sites should allow the main site and include an absolute sitemap URL. Publisher sites can disallow operational paths such as API, admin, account, cart, or checkout routes when those URLs should not use crawl budget.
After publishing, open /robots.txt in a browser, confirm the file is plain text, then run the Sitemap Checker and Search Console URL Inspection on important pages. Do not block the whole production site unless that is intentional.
Recommended Pattern For Small Tool Sites
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlThis pattern keeps public content crawlable while avoiding irrelevant operational paths. It does not replace authentication, noindex tags, canonical tags, or server-side access control.
Limitations
Robots.txt is a crawl instruction, not a content quality signal and not a guarantee of indexing control. A valid file can still coexist with ranking drops, indexing delays, duplicate canonical signals, or thin content issues. Treat this as one technical SEO guardrail.
Related Tools
Use the Sitemap Checker to confirm sitemap discovery, the GSC Export Planner once Search Console data arrives, and the GitHub PR SEO Checklist before merging route or metadata changes.
FAQ
What does robots.txt do?
A robots.txt file gives crawler access instructions for paths on a site. It can help keep crawl queues focused, but it is not a privacy or security system.
Can robots.txt stop a page from being indexed?
Not reliably. A blocked URL can still be discovered from links. Use noindex on crawlable pages when the goal is to keep a page out of search results.
Should I add a Sitemap line?
Yes for most public sites. A Sitemap line helps crawlers discover the canonical sitemap URL, but it does not guarantee crawling or indexing.
Should I use crawl-delay?
Avoid depending on crawl-delay for Google. Use Search Console crawl tools and server reliability work instead of assuming all crawlers support the same directive.
Can I block AI crawlers with robots.txt?
Some crawlers may respect explicit user-agent rules, but support varies. For RedBit-style publisher sites, do not rely on robots.txt alone for content licensing, privacy, or abuse prevention.