GrowingAlgo SEO Tools · Attract
Robots.txt Generator
A robots.txt file tells crawlers what they may fetch. You allow or disallow paths. A wrong Disallow: / can block the whole site. Use this so the file is clear and the sitemap line is included.
The tool
Try it here
Use this when you are writing or editing robots.txt. Check twice before you disallow. Staging and admin are normal to block. The public site usually is not.
Why it helps
Why you need a Robots.txt Generator
robots.txt is fetched first. If you disallow / , most crawlers will not fetch anything else. That is a common launch mistake on a leftover staging rule.
Disallow is not the same as noindex. Blocking a URL in robots.txt can hide it from Google even seeing the noindex tag. If you want a page out of the index, noindex it and still allow the crawl, or use Search Console removals with care.
Put the sitemap URL in robots.txt so crawlers can find the list of pages you want crawled.
When
When to use it
- At launch, when copying config from staging.
- When you add /admin/, /cart/, or search-filter paths you do not want crawled.
- When the sitemap path changes.
- When organic traffic drops suddenly and you need to rule out a robots block.
How
How to use it
- 1Start with User-agent: * and Allow: / unless you have a reason not to.
- 2Disallow specific paths only: /admin/, /thank-you/, /staging/. Leading slashes. No wild guess that blocks /de or /demo.
- 3Add Sitemap: https://acme.com/sitemap.xml (your real sitemap URL).
- 4Copy the file to the site root: https://acme.com/robots.txt.
- 5Open that URL in a browser. Confirm you did not disallow the whole site. Then test in Search Console’s robots tester if you can.
Example
Example: Acme robots.txt that does not block /demo
Acme must keep /demo crawlable (google-in-demo-exact-2026q3 and organic both use it). Admin and thank-you should stay out.
You put in
- Allow
- /
- Disallow
- /admin/, /thank-you/
- Sitemap
- https://acme.com/sitemap.xml
You get
User-agent: * Allow: / Disallow: /admin/ Disallow: /thank-you/ Sitemap: https://acme.com/sitemap.xml
Crawlers may fetch /demo. They should skip /admin/ and /thank-you/. There is no Disallow: / . If staging had that rule and someone copied it to production, search would go quiet.
Result
What this changes for you
Google can crawl the pages you care about. You avoid an accidental site-wide block, and crawlers know where the sitemap is.
Common mistakes
- Disallow: / on production. That blocks the whole site for most crawlers.
- Disallow: /demo when /demo is a money page. Check the path. /de and /demo are easy to confuse in a rush.
- Blocking CSS or JS that Google needs to render the page. If the page looks empty to the crawler, that can hurt.
- No sitemap line, and the sitemap living in a random folder nobody submitted.
FAQ
Robots.txt Generator FAQ
What is a robots.txt file?
robots.txt is a text file at your site root that tells crawlers which paths they may fetch. It is not a full security tool. It is a crawl instruction.
What is the difference between robots.txt Disallow and noindex?
Disallow asks crawlers not to fetch a URL. noindex asks them not to show it in search after they fetch it. If you Disallow a URL, Google may never see a noindex tag on that page.
Can robots.txt block my whole site?
Yes. User-agent: * plus Disallow: / blocks most crawlers from the whole site. Use Allow: / on production and only disallow specific folders.
Should robots.txt include a sitemap?
Yes. Add a Sitemap: line with the full URL of your XML sitemap so crawlers can find the list of pages you want crawled.
Where should robots.txt live?
At the site root: https://example.com/robots.txt. A file in a subfolder will not work as the site-wide robots file.