Should I allow GPTBot in robots.txt?

Yes, if you want ChatGPT to remember you exist. Add User-agent: GPTBot with Allow under it, named explicitly. Blocking GPTBot removes you from the model's future memory, and whatever you decide about training, never block ChatGPT-User or OAI-SearchBot.

Should I block CCBot?

Almost never. CCBot feeds Common Crawl, the public dataset much of the research world trains on. Blocking it does not stop one company, it quietly removes you from the corpus that teaches new models what the web contains.

Does robots.txt actually stop AI training?

It is compliance, not enforcement. The major labs honor the tokens, including GPTBot and Google-Extended, which is exactly why an explicit welcome works. If you need hard protection, pair robots.txt with terms of service and access controls.

The 2026 robots.txt: Roll Out the Red Carpet for AI

¶ CONTEXTTHE DOOR POLICY

01Yesterday you read the log. Today you write the policy.

The fashionable move of 2026 is blocking AI. Angry threads, one click firewalls, entire CDNs shipping AI blocking switched on by default for new sites. The owners find out months later, usually while asking why no model ever mentions them.

Half the sites missing from AI answers did not lose a fight. They locked the door and forgot.

The blocking wave has understandable roots. Publishers spent years getting scraped without a receipt, and the anger is fair. But the answer engines changed the deal. They cite, they link, they refer real people. Punishing the referrer for the scraper’s sins costs you the only distribution channel that is still growing.

I went the other way. Across all 21 of my domains, the door policy is an explicit welcome, written in the oldest file on the web. And because I measure citations instead of guessing, I can say it plainly: blocking AI crawlers in 2026 is deindexing yourself from the future.

robots.txt used to be plumbing. It is now the most consequential twenty lines on your server, because it decides whether the machines that answer questions are allowed to know you exist.

¶ CLAIMTHE VIP LIST

02Name your guests. Silence is not a welcome.

My file opens with a VIP section that names every major AI reader one by one: OpenAI’s three (GPTBot, ChatGPT-User, OAI-SearchBot), Anthropic’s ClaudeBot, Google-Extended for Gemini training, PerplexityBot, Applebot-Extended, Meta’s crawler, and CCBot.

Why name them when a wildcard already allows everyone? Because the two signals read differently. A wildcard allow is silence. Naming a bot is consent it can read. An explicit entry tells each lab your site treats it as a guest, not a trespasser it forgot to stop.

CCBot deserves a special word. It feeds Common Crawl, the public dataset half the research world trains on. Sites block it thinking they are blocking one company, and quietly exit the corpus that teaches new models what the web contains.

The red carpet is not an open house. The same file disallows my private paths for everyone, VIPs included. A CV that is nobody’s business, internal tools, drafts. Welcoming readers and guarding rooms are not in conflict. They are the same policy, written clearly.

The measured result of that open door: more than a million machine citations across those 21 domains in six months. The sites that welcomed the readers became the sites the readers quote.

EXTRACTED — THE SENTENCE THIS ENTRY EXISTS FOR

You cannot be quoted through a locked door.

¶ PLAYBOOKWRITE YOURS TONIGHT

03The red carpet, line by line

Here is the starter. Copy it, then adapt the private paths and the sitemap to your site.

// robots.txt — the red carpet starter, adapt the private paths

# THE VIP LIST — name every AI reader you welcome
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: Applebot-Extended
User-agent: CCBot
Allow: /

# EVERYONE, INCLUDING VIPS: private stays private
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

# HAND THEM THE MAP
Sitemap: https://yoursite.com/sitemap.xml

[1]

Name the VIPs explicitly. Every bot listed by its real user agent, then one shared Allow. The names are the message.

[2]

Guard the private rooms. Disallow admin panels, drafts, and personal files for everyone. A red carpet with locked side doors is a policy, not a contradiction.

[3]

Decide the training question consciously. If protecting your content from training matters more than future memory, block GPTBot and ClaudeBot and accept the cost. But never block ChatGPT-User or OAI-SearchBot. That is not protection, that is vanishing from live answers.

[4]

Check what your CDN decided for you. Open your firewall and bot settings tonight. If an AI blocking toggle shipped enabled, your beautiful robots.txt is a welcome sign on a locked door.

[5]

Hand them the map. End the file with your sitemap line, so every welcomed reader knows where everything lives.

One honest note: robots.txt is compliance, not enforcement. The major labs honor it, which is exactly why the welcome works. Your robots.txt is the first sentence of your reputation with machines. Tomorrow: why models lift sentences, not paragraphs, and how to write lines built for lifting.