Your website may be blocking AI tools without you knowing. If it is, ChatGPT, Claude, Perplexity and others can't read your pages, which makes it much harder for them to recommend you. You can check by looking at a single file on your site called robots.txt, and by asking your web host whether they block AI bots by default.

This is one of the most common problems we find, and one of the quickest to fix.

What is an AI crawler?

A crawler is an automated program that visits websites and reads their content. Google has used one for decades to build its search index. AI companies now run their own crawlers, which they use for two different jobs:

• Training, where content helps teach the AI model in general.

• Search and answers, where the AI fetches your page in real time to answer a specific question a user has asked.

The second job is the one that matters most for being recommended. If an AI tool can't fetch your page when a customer asks about your type of business, it has to rely on whatever it can find elsewhere, which may be out of date, incomplete or about a competitor.

The main AI crawlers to know

How to check your robots.txt file

Every website can have a file at yourdomain.com/robots.txt that tells crawlers what they may and may not read. Type that address into your browser.

You're looking for lines like these:

That means "GPTBot may not read any page on this site". If you see Disallow: / next to any of the search crawlers in the table above, or next to User-agent: * (which means every crawler), your site is blocking them.

If the file is empty, or you see Allow: /, the crawlers are generally permitted.

The hidden blocker: your web host or security service

Your robots.txt file can look perfectly fine while AI crawlers are still being turned away. That's because many sites sit behind a security or performance service, such as Cloudflare, which can block bots before they reach your site at all.

In July 2025, Cloudflare began blocking AI crawlers by default for new sites. Many hosting providers and website builders have similar settings. If your site uses one of these services, check its bot or AI crawler settings, or ask whoever manages your site to check for you.

Should you allow every AI crawler?

Not necessarily. It's a reasonable business decision to block training crawlers while allowing search crawlers. That way, AI tools can read your pages to answer customer questions, but your content isn't used to train future models.

For most service businesses, though, the priority is simple: make sure the search crawlers can get in. Those are the ones that decide whether you're part of the answer.

What to do next

1. Open yourdomain.com/robots.txt and check for any Disallow lines affecting AI crawlers.

2. Ask your web developer or host whether a firewall or bot-blocking setting is turning AI crawlers away.

3. If you find a block you didn't intend, ask for the search crawlers to be allowed.

4. Re-test in a month by asking ChatGPT or Perplexity a question your website clearly answers, and see whether it cites you.

Frequently asked questions

Will allowing AI crawlers slow my website down? For a typical business website, no. Crawlers visit occasionally and read a small number of pages.

Does blocking GPTBot stop ChatGPT recommending me? Not on its own, because GPTBot is used for training. But many sites that block GPTBot also block OAI-SearchBot by mistake, and that does affect ChatGPT search.

I use Wix, Squarespace or WordPress. Am I affected? Possibly. Each platform handles robots.txt and bot settings differently, and some add their own defaults. Check the robots.txt file directly rather than assuming.

Our free AI Visibility audit checks crawler access as standard, including blocks at the firewall level that a robots.txt check alone won't catch.