Robots.txt Generator
Robots.txt Generator
Use this free robots.txt generator to build a correct robots.txt file for your site, with presets for WordPress and Shopify, options for AI crawlers and your sitemap location.
Platform preset
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://www.example.com/sitemap.xml
Need a full crawl review? See our technical SEO services or pricing.
How it works
Three steps, under a minute
Pick your platform
Start from the WordPress, Shopify or custom preset.
Add your rules
Enter your sitemap URL, any extra paths to block and your choice on AI crawlers.
Copy, upload and test
Copy or download the file, put it at your domain root and check it in Search Console.
Reading your results
What the output means
- User-agentWhich crawler the rules below it apply to. An asterisk means every crawler.
- Disallow and AllowPaths crawlers should skip, with exceptions such as admin-ajax.php that themes and plugins need.
- SitemapWhere crawlers can find your XML sitemap, written as a full URL.
What a robots.txt file does
A robots.txt file sits at the root of your domain, for example example.com/robots.txt, and tells search engine crawlers which parts of your site they may crawl. It is mainly used to stop crawlers wasting time on low-value URLs such as internal search results, filter combinations, carts and admin areas.
Robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other pages link to it. To keep a page out of search results, allow it to be crawled and use a noindex meta tag instead.
WordPress and Shopify presets
The WordPress preset blocks the admin area while allowing admin-ajax.php, which many themes and plugins need, and blocks internal search results. Shopify generates its own robots.txt automatically, and you can only change it by editing the robots.txt.liquid template in your theme. The Shopify preset shows rules commonly added there, such as blocking extra filter and sort parameters.
Always test changes before going live. Search Console’s robots.txt report shows the version Google last fetched and any errors. If you are unsure, our technical SEO services include robots.txt and crawl budget reviews.
Common robots.txt mistakes
- Blocking the whole site with "Disallow: /" after a staging launch
- Blocking CSS and JavaScript files that Google needs to render pages
- Using robots.txt to hide pages you want removed from search results
- Forgetting that rules are case-sensitive and path-based
- Leaving out the sitemap line
01Where do I upload my robots.txt file?+
Upload it to the root of your domain so it is available at yourdomain.com/robots.txt. On WordPress, many SEO plugins let you edit it from the dashboard. On Shopify, edit the robots.txt.liquid theme template.
02Does robots.txt stop pages appearing in Google?+
Not reliably. It stops crawling, but a blocked page can still be indexed if other pages link to it. Use a noindex tag to keep pages out of search results.
03Should I block AI crawlers in robots.txt?+
It is a business decision. Blocking AI training crawlers such as GPTBot or Google-Extended can limit use of your content for training, but may also reduce visibility in some AI products. The generator lets you choose.
04Do I need a robots.txt file?+
It is not required, but it is good practice. Without one, crawlers assume they can crawl everything.
05How do I test my robots.txt file?+
Use the robots.txt report in Google Search Console to see the file Google fetched and any errors, and the URL Inspection tool to check whether a specific URL is blocked.
Want it done for you?
Get expert help with the work behind the numbers.
The tool gives you a starting point. Our technical seo services turn it into a prioritised plan and ship the work.
Prefer to hire through Upwork? Hire me on Upwork