Is your rental website blocking ChatGPT, Claude and other AI assistants?

Updated October 18, 2026 · Threshold

If an assistant's crawler is turned away from your site, that assistant can't read your pages, so it can't describe or recommend your place from them. It happens more often than hosts think, usually by accident.

The quick check

Open yoursite.com/robots.txt in a browser. If you see User-agent: * followed by Disallow: /, every automated reader is turned away, assistants included. Then look for any of these names followed by Disallow:

• ChatGPT: GPTBot, OAI-SearchBot, ChatGPT-User
• Claude: ClaudeBot, Claude-SearchBot, Claude-User
• Perplexity: PerplexityBot, Perplexity-User
• Gemini: Google-Extended

Or let our free website check read it for you: it lists which assistants your site turns away.

Settings that block assistants without you knowing

• Content delivery networks and security services often have a “block AI crawlers” or “AI bot” switch. Some providers turn it on by default for new sites.
• Website builders sometimes add AI crawlers to robots.txt through a privacy or “AI training” setting.
• A “noindex” tag left on from when the site was being built keeps search engines, and so assistants' searches, away.

Training versus search

Some names are for collecting text to train AI models; others are for answering a traveler's question right now. If you don't want your text used for training, you can block the training crawlers (such as GPTBot and Google-Extended) and still allow the search and user crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, Perplexity-User). The assistants then can't learn from your pages ahead of time, but they can still read them when a traveler asks.

The fix

Remove the lines that disallow the crawlers you want, or add a group that allows them, for example:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

Then turn off any “block AI bots” switch in your hosting or security settings. Changes take effect the next time each crawler visits.

Next: the full guide to being discoverable by AI assistants.