Free professional SEO tools — SERP preview, meta analyser, schema generator, robots.txt tester, and more. No login required.
A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or directories they can and cannot access. Important: robots.txt controls crawler access, not indexation. A page blocked by robots.txt won't be crawled, but it can still appear in search results if other pages link to it.
Block pages that serve no SEO value and consume crawl budget: admin areas (/admin/, /wp-admin/), staging paths, internal search result pages, checkout and cart pages, user account pages, duplicate filtered pages and thank-you pages. Never block CSS and JS files — Google needs them to render and evaluate your pages. Never block your sitemap.
Yes. You can block specific AI training crawlers by targeting their User-agent strings: GPTBot (OpenAI), Google-Extended (Gemini training), ClaudeBot (Anthropic), PerplexityBot and CCBot (Common Crawl). These directives only block crawling — they do not remove existing training data.
Incorrectly configured robots.txt can severely damage rankings. The most common mistakes are: accidentally blocking CSS/JS files (preventing Google from rendering pages), blocking your entire site with 'Disallow: /', or blocking important category or product pages. Always test your robots.txt using our Robots Tester after making changes.