Block AI crawlers, or put them to work?
GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider: AI bots crawl your site whether you invited them or not. Some bring citations and customers, others just take content or hammer your server. We help you choose per bot, and make it stick.
A choice you did not ask for, but have to make
Citations only flow to sites the bots can read
Some AI bots send you customers
When ChatGPT, Perplexity or Google AI Overviews answer a buying question, they cite sources, and cited businesses get named, trusted and clicked. Those answers can only include sites their crawlers are allowed to read. Block everything on principle and you have opted out of a channel your competitors are quietly benefiting from. If you want to be found this way, blocking is the wrong tool: AI search visibility work, including a proper llms.txt file, is the right one.
- Assistants cite and link the sources behind their answers
- Crawler access is a precondition for appearing in AI answers
- Accidental blocks are common and completely invisible to you
Robots.txt asks politely. Server rules enforce.
Others take your content and give nothing back
CCBot feeds Common Crawl, an open dataset used to train many AI models: your content goes in, no visitor ever comes out. Bytespider, ByteDance's crawler, has a reputation for aggressive request rates and for ignoring robots.txt entirely. On a small server, heavy bot traffic wastes bandwidth and can slow the site your real customers are using. For these, polite requests are not enough; blocking belongs at the server level, alongside your wider website security hardening.
- Training scrapers return no citations, traffic or credit
- Aggressive crawlers consume bandwidth and server capacity
- Bots that ignore robots.txt must be stopped at the server
Which AI crawlers to block, and which to welcome
These are the AI crawlers that matter most in UK server logs right now, and where we usually start the conversation. Your policy may differ, and that is the point: the recommendation should follow your business model, not a fashion.
| Bot name | Run by | What it feeds | Our default advice |
|---|---|---|---|
| GPTBot | OpenAI | Training data for future GPT models | Allow if AI visibility matters to you; block if your content is your product |
| OAI-SearchBot | OpenAI | ChatGPT search answers, with links to sources | Allow: this is the one that produces citations and clicks |
| ClaudeBot | Anthropic | Training and retrieval for the Claude assistant | Allow for most service businesses |
| PerplexityBot | Perplexity | Perplexity's answer engine, which cites sources prominently | Allow: strong citation behaviour |
| Google-Extended | Gemini model training (a robots.txt token, not a separate crawler) | Your call: blocking it does not affect Google Search rankings | |
| CCBot | Common Crawl | An open web dataset used to train many AI models | Block if you want control: it sends no traffic back |
| Bytespider | ByteDance (TikTok) | ByteDance model training; known for aggressive crawling | Block at server level: it is widely reported to ignore robots.txt |
Two things to hold onto. First, robots.txt is voluntary: it works on the well-behaved bots and does nothing about the rest, so meaningful blocks need server-level enforcement too. Second, and critically, Googlebot is not an AI crawler you tune. One careless rule aimed at AI bots can catch Googlebot and remove you from the search results that pay your bills. Every change we make is tested against Googlebot and Bingbot before and after it goes live.
From no policy to a deliberate one
Decide with you, not for you
We read your server logs to see which bots are already crawling and how hard, then talk through your model. A firm that sells expertise wants citing assistants in; a publisher whose words are the product may want most of them out. The policy follows the business.
Implement in two layers
Per-bot robots.txt directives handle the honest crawlers. Server-level user-agent and IP rules handle the ones that ignore requests. Googlebot and Bingbot access is verified before and after, so your normal SEO is never collateral damage.
Monitor and adjust
New AI crawlers appear all the time, and existing ones change behaviour. On an ongoing plan we track the bot share of your traffic, flag newcomers, and adjust the rules so the policy you chose stays the policy you have.
How visible is your website on Google right now?
Run our free SEO checker. It scans any page in about 30 seconds and shows the exact issues holding back your rankings.
Set-up and ongoing options
Crawler policy is included in the AI Search Visibility Package, managed month to month under Hosting Support, or scoped as part of a Full SEO Audit. Prices exclude VAT.
AI Search Visibility Package
Get your business cited by ChatGPT, Gemini, Perplexity and Google AI Overviews.
£399 one-off
- AI visibility audit: how assistants see you today
- llms.txt created and published
- Structured data upgraded site-wide
- Content tuned for AI citation
- AI crawler access policy configured
- Before and after visibility report
Hosting Support
Keep your current host. We manage it: server updates, SSL, DNS, email and fixes.
£39 /month
£390 /year
- Works with any UK or global host
- SSL certificates managed and renewed
- DNS and domain management
- Server-level troubleshooting
- Email setup and deliverability basics
- Migration help when you outgrow your host
Full SEO Audit
Technical, content and competitor audit with a plan you can act on.
£299 one-off
- Full technical crawl of your site
- Keyword and content gap analysis
- Competitor benchmark
- Backlink profile review
- Prioritised 90-day action plan
- Free 30-minute results call
All prices exclude VAT. Cancel monthly plans any time. Secure card and Direct Debit payments powered by Stripe.
Frequently asked questions
Should I block AI crawlers from my website?
It depends on your business model, which is exactly why a blanket answer is wrong. If you sell services or products, assistants that cite sources can send you customers, so blocking everything makes you invisible in AI answers. If your content is your product, training scrapers take value and return nothing. Most businesses land on a mixed policy: allow the citing assistants, block the pure scrapers.
How do I block GPTBot in robots.txt?
Add two lines to your robots.txt file: "User-agent: GPTBot" followed by "Disallow: /". That asks OpenAI's training crawler to stay away from the whole site. Remember robots.txt is a request, not a lock: reputable operators honour it, but badly behaved bots do not, which is why we back important blocks with server-level rules as well.
Will blocking AI crawlers hurt my Google rankings?
Not if it is done correctly. Google-Extended, the token that controls Gemini training use, is separate from Googlebot, so blocking it has no effect on Search. The danger is sloppy edits: a stray wildcard or a rule that catches Googlebot can wipe your site from Google. We verify Googlebot and Bingbot access before and after every change we make.
What is Google-Extended?
Google-Extended is a robots.txt token, not a crawler you will see in your logs. Adding a disallow rule for it tells Google not to use your content for training Gemini and related AI models. Google has confirmed it does not affect how pages rank in Search, so it is a genuine opt-out you can use without an SEO penalty.
Do AI bots ignore robots.txt?
Some do. Reputable operators such as OpenAI, Anthropic and Google publish their user agents and honour robots.txt. Bytespider, ByteDance's crawler, has been widely reported ignoring it, and plenty of anonymous scrapers spoof normal browser user agents. That is why a real policy has two layers: robots.txt for the honest bots, server-level user-agent and IP rules for the rest.
How do I know which bots are crawling my site?
Your server access logs record every request with its user agent, and that is where we start: measuring what share of your traffic is AI bots, which ones, and what they are fetching. Guessing is pointless because the picture varies hugely between sites. On our monitoring plans we review bot traffic regularly and flag new crawlers as they appear.