Dataset

Which AI crawlers 1,020 small ad-funded publishers name and block in robots.txt

Historical robots.txt policies from 1,020 readable files in a sample drawn from a 16,076-domain advertising-network frame. Named tokens, resolved site-root blocks, legacy-name flags and template markers are separate fields. In the saved snapshot, 244 of them named evaluated crawlers: 216 blocked at least one at the root and 28 blocked none. Of the 244, 167 also carried legacy-name flags. These observations describe public file contents, not owner intent, actual crawler compliance, lost referrals, model training or licensing income. The source was collected on 2026-09-02; descriptions and denominators were clarified on 2026-10-20.

• 16,076 domains in the frame
• 1,200 domains drawn
• 1,020 readable files
• 9 AI crawlers resolved against every one

Coverage

2026-09-02

Last updated

2026-10-20

Licence

CC BY 4.0

DOI

10.5281/zenodo.22280370

What a row is

One publisher’s robots.txt file: which AI crawlers it names, which it actually blocks, which names it carries that no vendor documents, and whose template it was written from.

Subjects

robots.txt, AI crawlers, GPTBot, ChatGPT-User, ClaudeBot, CCBot, Google-Extended, crawler blocking, AI training opt-out, web publishers, RFC 9309

Download the public data sample1,020 rows · 8 columns · CSV · no account, no email

Read the finding this came out ofYour AI crawler rules may block a visitor you wanted.

How to cite this

Use it commercially, change it, build on it. The only condition is credit, and a link satisfies it.

What you should know about this data

• These are small, advertising-funded content businesses on two advertising networks. They are not the web. A publisher with no ad network is not in the frame at all, and the large publishers that every other published figure comes from behave completely differently.
• A file that could not be fetched is a fact about a firewall, not a policy. The 180 domains that refused are left out of the denominator rather than counted as silent.
• The categories are not a partition and must not be added up. A file that blocks a crawler which would have cited it also blocks at least one crawler, and often carries names somebody else’s template wrote. Adding them gives a number larger than the sample.
• “No vendor documents it” is not “it is dead”. We checked each name against its supposed operator’s own published crawler list. Anthropic documents ClaudeBot, Claude-User and Claude-SearchBot, and neither anthropic-ai nor claude-web is among them — but no vendor publishes a retirement notice, so what we can show is that a publisher is naming something the operator does not list, not that nothing answers to it.
• A vendor marker says part of the file was not written by the publisher. It does not say the AI rules in particular came from the vendor — a Yoast block commonly contains no AI rules at all.
• Domains are omitted from the download. The published frame, seed and method allow the sample to be redrawn; omission is not a guarantee against re-identification.

How it was measured

1,200 domains were drawn once from the 16,076 the two networks publish about themselves, ranked by a hash of a published seed and the domain name, so that anyone can redraw the same sample. The nightly crawler fetched each robots.txt on a single complete pass; 1,020 answered and 180 refused or could not be reached, and the ones that could not be reached are excluded from the denominator rather than counted as saying nothing, which would have moved the headline in the direction that flatters it. Each file was then resolved rather than searched: the rules that reach a named crawler at the site root, worked out by the grouping rules the robots.txt standard sets out, because a file that stacks thirty crawler names above one permissive block has authorised all thirty. Rows are in the order the source holds them, which is alphabetical by domain with the domain removed — unrelated to anything measured, so any run of rows from it is a fair slice of the whole.

The summary

The first 60 of 1,020 rows. The download has all of them. Free to use with credit, commercially included.

What each column means

network

The advertising network whose published customer list the domain was drawn from

names_crawlers

How many current AI crawlers the file names at all, whether or not it blocks them

blocks_crawlers

How many of them it actually blocks at the site root, once the file is resolved

named_tokens

The crawlers the file names, separated by semicolons

blocked_tokens

The crawlers the file blocks, separated by semicolons

blocks_citation_crawler

yes when at least one blocked crawler is one that fetches a page to answer a question and links back

retired_tokens

Names in the file that no vendor currently documents, separated by semicolons. Not a claim that the crawler stopped running — only that its supposed operator does not list it

vendor_template

The tool or network whose marked block appears in the file, where one does

← Every dataset