Generate a robots.txt file for your website. Control which pages search engine bots can crawl. Add sitemap URL and configure per-bot rules.
Robots.txt Generator
Default Bot Rules
Disallow These Paths (one per line)
Block Specific Bots
Ad · 336x280
Robots.txt: What It Is and How to Configure It
Robots.txt is a text file at the root of your domain (yoursite.com/robots.txt) that tells search engine crawlers which pages they can and cannot access. It's the first thing Googlebot checks before crawling your site.
Robots.txt Syntax
User-agent: * — Applies to all bots (Google, Bing, etc.)
Disallow: /path/ — Block this path for the specified user-agent
Allow: /path/ — Override a Disallow rule for a specific path
Sitemap: https://example.com/sitemap.xml — Tell search engines where your sitemap is
Crawl-delay: 1 — Ask bots to wait 1 second between requests (reduces server load)
What to Block with Robots.txt
/admin/, /login/, /account/ — Private pages that shouldn't appear in search
/cart/, /checkout/ — Transactional pages with no SEO value
/api/ — API endpoints that return JSON, not HTML
/search? — Site search result pages (thin, duplicate content)
/wp-admin/ — WordPress admin area (security + crawl budget)
Important: Robots.txt is Not Security
Robots.txt tells cooperative bots not to crawl. Malicious scrapers ignore it. For truly sensitive pages, require authentication. Also: blocking a URL in robots.txt doesn't remove it from Google's index — use noindex meta tag for that.
Frequently Asked Questions
The robots.txt file must be in the root directory of your website, accessible at https://yourdomain.com/robots.txt. It cannot be in a subdirectory — /blog/robots.txt only covers the /blog/ path, not the whole domain. For a WordPress site, most hosting control panels let you edit it directly, or upload via FTP to the public_html folder.
No — robots.txt prevents Googlebot from crawling the URL, but Google may still index it if other sites link to it. To remove a URL from Google's index: (1) add noindex meta tag in the page's HTML, (2) use Google Search Console's URL Removal Tool for quick removal, (3) return a 404 or 410 status code. The noindex tag is more reliable than robots.txt for de-indexing.
It's a legitimate choice if you don't want your content used to train AI models. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl used by many LLMs), and Google-Extended (Bard) all respect robots.txt Disallow rules. However, blocking these bots won't remove content already in their training data. For new content, blocking is effective; for existing content, it prevents future scraping only.
Robots.txt controls crawling (whether bots visit the URL). Meta robots controls indexing (whether bots index the page content): . A page blocked by robots.txt can still be indexed if linked from elsewhere — Google sees the URL without crawling the content. For complete control: allow crawling (no robots.txt block) but add noindex meta tag, so Google can read the tag and know not to index it.