Presets:
User-Agent Groups
Sitemap & Crawl Settings
Live Preview
⚡
Live Preview
See your robots.txt update in real-time as you add rules and directives.
🛡️
Smart Validation
Detects conflicting rules, missing newlines and common mistakes instantly.
🤖
Block AI Bots
One-click preset to block ChatGPT, GPTBot, CCBot, Google-Extended and Bytespider.
📦
Ready Presets
Pre-built templates for WordPress, Shopify, allow-all and block-all configurations.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file tells search engine crawlers which pages or sections of your site they can or cannot crawl. It is placed at the root of your domain (e.g. https://example.com/robots.txt).
Where should I place my robots.txt file?
Upload it to the root directory of your website so it is accessible at /robots.txt. For example, https://example.com/robots.txt. Most CMS platforms allow you to upload or edit it via their file manager.
Does robots.txt block pages from appearing in search results?
No. robots.txt controls crawling, not indexing. A page disallowed in robots.txt can still appear in search results if linked from other sites. Use noindex meta tags for pages you want excluded from indexes.